Jul 12 17:01:07.180659 ip-10-0-139-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 12 17:01:07.180672 ip-10-0-139-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 12 17:01:07.180683 ip-10-0-139-89 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 12 17:01:07.181059 ip-10-0-139-89 systemd[1]: Failed to start Kubernetes Kubelet. Jul 12 17:01:17.256512 ip-10-0-139-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 12 17:01:17.256527 ip-10-0-139-89 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d8414e91720546889454e8ccb873c0b5 -- Jul 12 17:03:10.891595 ip-10-0-139-89 systemd[1]: Starting Kubernetes Kubelet... Jul 12 17:03:11.233416 ip-10-0-139-89 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 17:03:11.233416 ip-10-0-139-89 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 17:03:11.233416 ip-10-0-139-89 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 17:03:11.233416 ip-10-0-139-89 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 17:03:11.233416 ip-10-0-139-89 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 17:03:11.236571 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.234227 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243489 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243504 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243509 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243514 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243519 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243523 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243527 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243532 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243536 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243540 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243544 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243553 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243557 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243561 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243565 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243569 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243583 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243588 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243592 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 17:03:11.246775 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243595 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243600 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243606 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243610 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243615 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243623 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243626 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243630 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243634 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243638 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243642 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243647 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243651 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243657 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243663 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243667 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243672 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243681 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243686 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 17:03:11.247436 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243690 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243694 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243699 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243703 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243707 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243712 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243717 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243723 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243727 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243731 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243735 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243758 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243763 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243767 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243771 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243776 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243780 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243784 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243790 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243795 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 17:03:11.249463 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243799 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243806 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243815 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243819 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243823 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243829 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243833 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243837 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243842 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243846 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243850 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243854 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243858 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243868 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243872 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243876 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243880 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243884 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243889 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243892 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 17:03:11.336738 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243896 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 17:03:11.315223 ip-10-0-139-89 systemd[1]: Started Kubernetes Kubelet. Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243900 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243906 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243910 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243925 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243929 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243937 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243941 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243945 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243949 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243953 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243957 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243961 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243965 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243969 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243973 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243979 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243985 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243990 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 17:03:11.338904 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.243997 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.244001 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246582 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246597 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246601 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246605 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246610 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246614 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246618 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246623 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246627 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246632 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246636 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246639 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246646 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246656 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246660 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246665 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246681 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 17:03:11.339707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246686 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246690 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246695 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246699 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246703 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246707 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246712 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246716 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246720 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246728 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246732 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246737 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246741 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246746 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246750 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246754 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246760 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246764 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246768 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246773 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 17:03:11.340710 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246777 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246790 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246795 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246799 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246803 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246807 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246811 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246815 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246819 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246823 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246831 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246838 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246849 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246858 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246862 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246866 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246870 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246874 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246878 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 17:03:11.341551 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246882 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246886 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246890 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246894 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246898 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246902 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246907 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246915 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246920 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246924 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246927 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246932 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246937 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246941 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246945 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246950 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246954 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246959 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246970 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 17:03:11.346505 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246976 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246981 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246985 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246990 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246994 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.246998 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247002 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247013 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247017 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247021 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247025 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247033 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247037 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247042 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247046 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247050 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247054 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247058 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247063 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247067 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 17:03:11.347635 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247072 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247076 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247079 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.247083 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248163 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248173 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248179 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248185 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248189 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248194 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248202 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248206 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248210 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248215 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248219 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248223 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248227 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248231 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248235 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 17:03:11.348518 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248255 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248279 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248284 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248292 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248296 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248300 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248304 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248308 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248312 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248316 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248320 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248324 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248328 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248333 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248337 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248341 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248350 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248354 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248357 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248363 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 17:03:11.351594 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248367 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248371 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248375 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248379 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248383 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248387 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248392 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248401 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248405 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248409 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248413 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248416 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248420 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248432 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248436 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248440 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248446 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248450 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248455 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248459 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 17:03:11.352228 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248466 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248470 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248474 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248478 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248482 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248493 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248497 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248501 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248505 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248509 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248513 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248521 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248525 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248529 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248533 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248537 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248541 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248545 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248549 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 17:03:11.352858 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248553 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248557 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248561 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248567 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248576 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248584 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248587 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248599 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248604 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248608 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248612 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248616 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248620 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248624 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248627 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248631 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248635 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248640 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248657 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248661 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 17:03:11.354178 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248666 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248670 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248674 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248678 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.248682 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248815 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248824 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248835 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248841 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248848 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248853 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248864 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248871 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248875 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248880 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248885 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248890 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248900 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248905 2573 flags.go:64] FLAG: --cgroup-root="" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248913 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248925 2573 flags.go:64] FLAG: --client-ca-file="" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248930 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248935 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248944 2573 flags.go:64] FLAG: --cluster-domain="" Jul 12 17:03:11.354873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248948 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248953 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248958 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248964 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248973 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248978 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248986 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248991 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.248996 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249001 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249006 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249010 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249018 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249025 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249029 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249034 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249038 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249047 2573 flags.go:64] FLAG: --enable-server="true" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249052 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249066 2573 flags.go:64] FLAG: --event-burst="100" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249071 2573 flags.go:64] FLAG: --event-qps="50" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249080 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249084 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249089 2573 flags.go:64] FLAG: --eviction-hard="" Jul 12 17:03:13.007784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249094 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249099 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249104 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249108 2573 flags.go:64] FLAG: --eviction-soft="" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249112 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249128 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249133 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249138 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249142 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249146 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249151 2573 flags.go:64] FLAG: --feature-gates="" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249157 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249162 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249170 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249181 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249186 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249191 2573 flags.go:64] FLAG: --help="false" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249196 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249201 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249205 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249209 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249215 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249224 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249229 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 17:03:13.029794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249233 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249240 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249245 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249250 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249255 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249259 2573 flags.go:64] FLAG: --kube-reserved="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249280 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249285 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249290 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249295 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249299 2573 flags.go:64] FLAG: --lock-file="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249304 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249308 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249313 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249334 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249343 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249347 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249352 2573 flags.go:64] FLAG: --logging-format="text" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249357 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249362 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249367 2573 flags.go:64] FLAG: --manifest-url="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249372 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249390 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249399 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249405 2573 flags.go:64] FLAG: --max-pods="110" Jul 12 17:03:13.032392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249409 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249414 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249418 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249423 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249428 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249432 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249437 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249452 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249458 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249462 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249467 2573 flags.go:64] FLAG: --pod-cidr="" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249472 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249481 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249485 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249495 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249500 2573 flags.go:64] FLAG: --port="10250" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249505 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249509 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0045cd9e4e4912a23" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249515 2573 flags.go:64] FLAG: --qos-reserved="" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249519 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249524 2573 flags.go:64] FLAG: --register-node="true" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249528 2573 flags.go:64] FLAG: --register-with-taints="" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249546 2573 flags.go:64] FLAG: --registry-burst="10" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249552 2573 flags.go:64] FLAG: --registry-qps="5" Jul 12 17:03:13.033705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249556 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249560 2573 flags.go:64] FLAG: --reserved-memory="" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249566 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249571 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249576 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249581 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249586 2573 flags.go:64] FLAG: --runonce="false" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249594 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249599 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249603 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249608 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249612 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249617 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249621 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249627 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249635 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249639 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249645 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249650 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249655 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249659 2573 flags.go:64] FLAG: --system-cgroups="" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249663 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249676 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249680 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249684 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 17:03:13.187043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249721 2573 flags.go:64] FLAG: --tls-min-version="" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249727 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249732 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249737 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249742 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249750 2573 flags.go:64] FLAG: --v="2" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249763 2573 flags.go:64] FLAG: --version="false" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249770 2573 flags.go:64] FLAG: --vmodule="" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249775 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.249780 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250195 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250201 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250208 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250212 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250217 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250222 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250231 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250240 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250245 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250249 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250253 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 17:03:13.188500 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250257 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250283 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250288 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250305 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250309 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250313 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250317 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250321 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250326 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250334 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250339 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250352 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250356 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250360 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250364 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250368 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250372 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250376 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250388 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250393 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 17:03:13.189601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250397 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250400 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250409 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250413 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250419 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250423 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250427 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250431 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250435 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250440 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250443 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250450 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250456 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250460 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250469 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250475 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250480 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250484 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250488 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250492 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 17:03:13.191361 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250496 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250500 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250504 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250508 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250514 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250519 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250528 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250532 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250536 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250540 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250544 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250555 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250560 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250571 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250575 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250579 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250585 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250590 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250595 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 17:03:13.192455 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250603 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250606 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250610 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250614 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250618 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250622 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250626 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250630 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250635 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250639 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250643 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250647 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250654 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250658 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250663 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250667 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250671 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250675 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250679 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250683 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 17:03:13.193694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250687 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250691 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250697 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250703 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250711 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250723 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250727 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250731 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.250735 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.251398 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.258501 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.258517 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.258533 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.258539 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 17:03:13.194595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259047 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259054 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259252 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259257 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259469 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259667 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259671 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259674 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259677 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259679 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259875 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259877 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259880 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259883 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.259886 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260085 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260088 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260090 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260093 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 17:03:13.195493 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260300 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260496 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260499 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260503 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260506 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260704 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260707 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260710 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260714 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260716 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260722 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260725 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260728 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260731 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260733 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260736 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260739 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260741 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260744 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260747 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 17:03:13.197248 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260749 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260752 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260754 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260757 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260760 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260763 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260765 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260768 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260771 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260774 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260779 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260781 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260784 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260786 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260790 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260793 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260796 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260799 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260801 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260804 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 17:03:13.198063 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260807 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260809 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260811 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260814 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260817 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260819 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260822 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260824 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260827 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260830 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260833 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260837 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260840 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260843 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260847 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260852 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260854 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260857 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260860 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 17:03:13.198864 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260863 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260866 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260868 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260872 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260875 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260877 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260880 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260883 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260886 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260888 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260893 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260895 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260898 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260901 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260904 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260906 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260909 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260911 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260914 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260917 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 17:03:13.199603 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.260919 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.260924 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261065 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261071 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261074 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261077 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261080 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261083 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261086 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261088 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261091 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261094 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261096 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261099 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261102 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 17:03:13.200376 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261104 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261108 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261111 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261113 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261116 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261119 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261123 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261129 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261131 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261134 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261137 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261140 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261143 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261146 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261149 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261152 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261155 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261158 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261161 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261164 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 17:03:13.200955 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261166 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261169 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261172 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261175 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261177 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261180 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261183 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261186 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261188 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261191 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261193 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261196 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261199 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261203 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261207 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261210 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261212 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261227 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261231 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 17:03:13.201698 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261235 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261238 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261240 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261243 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261246 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261248 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261251 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261253 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261256 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261259 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261271 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261276 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261279 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261282 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261285 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261288 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261291 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261293 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261296 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 17:03:13.202409 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261299 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261301 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261304 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261306 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261309 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261312 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261315 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261320 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261322 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261325 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261328 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261331 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261333 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261337 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261340 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261342 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261345 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261348 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261350 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261353 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 17:03:13.203125 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261355 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261358 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261361 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261363 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261366 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261369 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261371 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:11.261374 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.261379 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.261509 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.263632 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.264804 2573 server.go:1019] "Starting client certificate rotation" Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.264900 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.264936 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 17:03:13.203939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.266997 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 17:03:13.204479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.268145 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 17:03:13.204479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.276848 2573 log.go:25] "Validated CRI v1 runtime API" Jul 12 17:03:13.204479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.279446 2573 log.go:25] "Validated CRI v1 image API" Jul 12 17:03:13.204479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.280136 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 17:03:13.204479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.288011 2573 fs.go:135] Filesystem UUIDs: map[6e624dc5-83e2-4682-bd94-9864e0ff54a5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b4f349e2-3aff-43a5-9ecd-939db3cd83d0:/dev/nvme0n1p4] Jul 12 17:03:13.204479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.288038 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.293718 2573 manager.go:216] Machine: {Timestamp:2026-07-12 17:03:11.292482122 +0000 UTC m=+0.300035691 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100472 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2503f17ac2f1c18c7efdaab19d5cfb SystemUUID:ec2503f1-7ac2-f1c1-8c7e-fdaab19d5cfb BootID:d8414e91-7205-4688-9454-e8ccb873c0b5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:61:b0:b9:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:61:b0:b9:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:4f:c8:29:36:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.293902 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.294494 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.297406 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.297441 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.298114 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.298126 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.298134 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.298710 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.299585 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.300377 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303514 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303682 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303715 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303728 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303743 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 17:03:13.204722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303760 2573 kubelet.go:398] "Adding apiserver pod source" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.303774 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.304239 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.304257 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.307398 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.308956 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.308984 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309756 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309770 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309776 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309782 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309790 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309797 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309810 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309817 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309825 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309831 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309840 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.309848 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.311048 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.311056 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.314364 2573 server.go:1268] "Started kubelet" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.314438 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.314465 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.314667 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.316448 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.316624 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.316664 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.316721 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.316847 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.320284 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.320294 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.320419 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.320922 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.320925 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.320943 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.321036 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.321047 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.321963 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ztkvt" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.321410 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-89.ec2.internal.18c19994b0b5b7ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-89.ec2.internal,UID:ip-10-0-139-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-89.ec2.internal,},FirstTimestamp:2026-07-12 17:03:11.314335727 +0000 UTC m=+0.321889296,LastTimestamp:2026-07-12 17:03:11.314335727 +0000 UTC m=+0.321889296,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-89.ec2.internal,}" Jul 12 17:03:13.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326618 2573 factory.go:153] Registering CRI-O factory Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.326612 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.326652 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326740 2573 factory.go:223] Registration of the crio container factory successfully Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326809 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326820 2573 factory.go:55] Registering systemd factory Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326830 2573 factory.go:223] Registration of the systemd container factory successfully Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326862 2573 factory.go:103] Registering Raw factory Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.326880 2573 manager.go:1196] Started watching for new ooms in manager Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.327742 2573 manager.go:349] Starting recovery of all containers Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.329427 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ztkvt" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.330003 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.337000 2573 manager.go:354] Recovery completed Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.341601 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.345361 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.345385 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.345394 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.345862 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.345869 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.345884 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.346156 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.347023 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.347044 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.347057 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.347678 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-89.ec2.internal.18c19994b28f551f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-89.ec2.internal,UID:ip-10-0-139-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-89.ec2.internal,},FirstTimestamp:2026-07-12 17:03:11.345374495 +0000 UTC m=+0.352928065,LastTimestamp:2026-07-12 17:03:11.345374495 +0000 UTC m=+0.352928065,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-89.ec2.internal,}" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.348024 2573 policy_none.go:49] "None policy: Start" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.348052 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.348067 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.350147 2573 policy_none.go:47] "Start" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.393152 2573 manager.go:337] "Starting Device Plugin manager" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.393325 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.393337 2573 server.go:87] "Starting device plugin registration server" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.393547 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.393555 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.393968 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.394042 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:11.394050 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.394645 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:11.394684 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.006927 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.206948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.008144 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.008171 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.008181 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.008209 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.014639 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.014661 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-89.ec2.internal\": node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.027777 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.032278 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.032310 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.032335 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.032371 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.034380 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.128257 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.133409 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal"] Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.133487 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.135204 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.135281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bafc608fa264eba1d850d33cd5b6db88-config\") pod \"kube-apiserver-proxy-ip-10-0-139-89.ec2.internal\" (UID: \"bafc608fa264eba1d850d33cd5b6db88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.139824 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.139841 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.142330 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.142422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.142452 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.143466 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.143494 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.143505 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.143466 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.143572 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.143585 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.145884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.145925 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.146660 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientMemory" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.146680 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.146693 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeHasSufficientPID" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.163544 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-89.ec2.internal\" not found" node="ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.208521 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.168203 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-89.ec2.internal\" not found" node="ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.228735 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.228705 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.236074 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.236051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bafc608fa264eba1d850d33cd5b6db88-config\") pod \"kube-apiserver-proxy-ip-10-0-139-89.ec2.internal\" (UID: \"bafc608fa264eba1d850d33cd5b6db88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.236138 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.236082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef8e334d27c2be36757cd606a2fea1d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal\" (UID: \"ef8e334d27c2be36757cd606a2fea1d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.236182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.236103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8e334d27c2be36757cd606a2fea1d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal\" (UID: \"ef8e334d27c2be36757cd606a2fea1d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.236182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.236163 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bafc608fa264eba1d850d33cd5b6db88-config\") pod \"kube-apiserver-proxy-ip-10-0-139-89.ec2.internal\" (UID: \"bafc608fa264eba1d850d33cd5b6db88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.329559 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.329517 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.336857 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.336833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef8e334d27c2be36757cd606a2fea1d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal\" (UID: \"ef8e334d27c2be36757cd606a2fea1d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.336929 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.336860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8e334d27c2be36757cd606a2fea1d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal\" (UID: \"ef8e334d27c2be36757cd606a2fea1d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.336929 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.336901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8e334d27c2be36757cd606a2fea1d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal\" (UID: \"ef8e334d27c2be36757cd606a2fea1d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.336991 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.336962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef8e334d27c2be36757cd606a2fea1d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal\" (UID: \"ef8e334d27c2be36757cd606a2fea1d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.430074 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.430034 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.466600 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.466545 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.471559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.471506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.530214 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.530174 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.630780 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.630745 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.723849 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.723775 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:13.731664 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.731634 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-89.ec2.internal\" not found" Jul 12 17:03:13.747290 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.747268 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:13.777074 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.777045 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 17:03:13.777228 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.777212 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 17:03:13.777277 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.777239 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 17:03:13.777277 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.777256 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 17:03:13.819394 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.819360 2573 apiserver.go:52] "Watching apiserver" Jul 12 17:03:13.833143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.833115 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 17:03:13.833274 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.833115 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.843039 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.842880 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 17:03:13.843378 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.843350 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 17:03:13.843516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.843421 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" Jul 12 17:03:13.843852 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.843832 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-58k2l","openshift-multus/multus-additional-cni-plugins-v79f6","openshift-network-diagnostics/network-check-target-zjsj4","kube-system/global-pull-secret-syncer-tw7w7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6","openshift-cluster-node-tuning-operator/tuned-dsp6f","openshift-dns/node-resolver-8jk4d","openshift-multus/network-metrics-daemon-clc6z","openshift-network-operator/iptables-alerter-wt8w7","openshift-ovn-kubernetes/ovnkube-node-dvmx2","kube-system/konnectivity-agent-4qprc","openshift-image-registry/node-ca-x48ln"] Jul 12 17:03:13.846444 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.846424 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.847307 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.847263 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 16:58:12 +0000 UTC" deadline="2028-03-04 02:48:08.894692342 +0000 UTC" Jul 12 17:03:13.847385 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.847308 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14409h44m55.047388284s" Jul 12 17:03:13.849953 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.849937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.850495 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.850475 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 17:03:13.850588 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.850505 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5f75z\"" Jul 12 17:03:13.850588 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.850507 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.850698 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.850679 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 17:03:13.850787 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.850770 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.850851 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.850820 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 17:03:13.852103 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.852083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:13.852184 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.852168 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:13.852316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.852300 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nxh5n\"" Jul 12 17:03:13.852374 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.852363 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 17:03:13.852514 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.852499 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 17:03:13.854459 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.854441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:13.854578 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.854501 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:13.855424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.855403 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 17:03:13.856568 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.856554 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.858630 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.858609 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 17:03:13.858730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.858693 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.858815 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.858797 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ssf9h\"" Jul 12 17:03:13.858996 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.858978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.859332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.859315 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.861172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.861155 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-shwc4\"" Jul 12 17:03:13.861242 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.861212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:13.861242 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.861237 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.861342 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.861309 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.863272 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.863255 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ts6sj\"" Jul 12 17:03:13.863475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.863426 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.863592 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.863498 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.863650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.863629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:13.863705 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:13.863688 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:13.865961 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.865846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:13.868223 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.868203 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 17:03:13.868293 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.868269 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.868376 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.868288 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.869167 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.869143 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.870563 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.870541 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qlwk5\"" Jul 12 17:03:13.871529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.871113 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxbn5\"" Jul 12 17:03:13.871529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.871362 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 17:03:13.871693 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.871630 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:13.872369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.872303 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5bj4" Jul 12 17:03:13.872369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.872302 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.872519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.872373 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.872519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.872465 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 17:03:13.873142 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.873122 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 17:03:13.873344 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.873331 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 17:03:13.874076 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.874060 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal"] Jul 12 17:03:13.874147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.874093 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal"] Jul 12 17:03:13.874181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.874150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:13.875194 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.875177 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 17:03:13.876312 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.876287 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hjnhd\"" Jul 12 17:03:13.876449 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.876364 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 17:03:13.876521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.876460 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 17:03:13.876605 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.876584 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 17:03:13.876856 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.876843 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6bhd6\"" Jul 12 17:03:13.876951 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.876877 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 17:03:13.879302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.879279 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5bj4" Jul 12 17:03:13.934686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.934668 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 17:03:13.940347 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00c5aa6e-c15c-4b8a-b201-44c4355ef991-serviceca\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:13.940454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-daemon-config\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.940454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-registration-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.940454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-host\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.940454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-env-overrides\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-var-lib-kubelet\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b559k\" (UniqueName: \"kubernetes.io/projected/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-kube-api-access-b559k\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-etc-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb942b28-ce95-4401-bb85-b25474302d61-ovn-node-metrics-cert\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-modprobe-d\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-host-slash\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:13.940694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-slash\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-ovn\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-node-log\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-log-socket\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-socket-dir-parent\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-netns\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-kubernetes\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-iptables-alerter-script\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:13.940943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00c5aa6e-c15c-4b8a-b201-44c4355ef991-host\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysctl-d\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940967 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-tmp-dir\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.940987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-systemd-units\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1811a7cf-f4d2-4424-a69d-aa8b9e402759-tmp\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-k8s-cni-cncf-io\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cnibin\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-sys-fs\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysconfig\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-systemd\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-cni-bin\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.941232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8px44\" (UniqueName: \"kubernetes.io/projected/cf74a844-f099-4a72-804e-0106c3b44a3c-kube-api-access-8px44\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-system-cni-dir\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-run\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-lib-modules\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-os-release\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-systemd\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-conf-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9pt\" (UniqueName: \"kubernetes.io/projected/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-kube-api-access-db9pt\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqp6\" (UniqueName: \"kubernetes.io/projected/1811a7cf-f4d2-4424-a69d-aa8b9e402759-kube-api-access-2tqp6\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-var-lib-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a35afdb-8438-4880-a499-ff342dee3802-agent-certs\") pod \"konnectivity-agent-4qprc\" (UID: \"8a35afdb-8438-4880-a499-ff342dee3802\") " pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-multus-certs\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cni-binary-copy\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-tuned\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.941609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a35afdb-8438-4880-a499-ff342dee3802-konnectivity-ca\") pod \"konnectivity-agent-4qprc\" (UID: \"8a35afdb-8438-4880-a499-ff342dee3802\") " pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf74a844-f099-4a72-804e-0106c3b44a3c-cni-binary-copy\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-cni-multus\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447h2\" (UniqueName: \"kubernetes.io/projected/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-kube-api-access-447h2\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-socket-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-hosts-file\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl89h\" (UniqueName: \"kubernetes.io/projected/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-kube-api-access-hl89h\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvm5k\" (UniqueName: \"kubernetes.io/projected/00c5aa6e-c15c-4b8a-b201-44c4355ef991-kube-api-access-tvm5k\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-kubelet\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-sys\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-kubelet\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.941954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-ovnkube-config\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-cnibin\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-os-release\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942042 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-hostroot\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef941278-2d82-4baa-bb36-caa777b09ab0-kubelet-config\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef941278-2d82-4baa-bb36-caa777b09ab0-dbus\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-device-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-ovnkube-script-lib\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zh9\" (UniqueName: \"kubernetes.io/projected/7115496d-2b24-4f09-b65a-678040e4a303-kube-api-access-99zh9\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-system-cni-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysctl-conf\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-cni-netd\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942270 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-cni-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-cni-bin\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmjv\" (UniqueName: \"kubernetes.io/projected/fb942b28-ce95-4401-bb85-b25474302d61-kube-api-access-lqmjv\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-etc-kubernetes\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:13.942650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-run-netns\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:13.943167 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:13.942423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.021264 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.021229 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8e334d27c2be36757cd606a2fea1d6.slice/crio-e8fc1beda29d329feec67ac17f7053eb7bbd37c50cbde292d7b52f3aaa1ca3d5 WatchSource:0}: Error finding container e8fc1beda29d329feec67ac17f7053eb7bbd37c50cbde292d7b52f3aaa1ca3d5: Status 404 returned error can't find the container with id e8fc1beda29d329feec67ac17f7053eb7bbd37c50cbde292d7b52f3aaa1ca3d5 Jul 12 17:03:14.029755 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.029737 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 17:03:14.035294 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.035270 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafc608fa264eba1d850d33cd5b6db88.slice/crio-75c24e12f18c5a0a3c48b1b94e3d88bbd956c1765d9c2d2ffc47858e36546391 WatchSource:0}: Error finding container 75c24e12f18c5a0a3c48b1b94e3d88bbd956c1765d9c2d2ffc47858e36546391: Status 404 returned error can't find the container with id 75c24e12f18c5a0a3c48b1b94e3d88bbd956c1765d9c2d2ffc47858e36546391 Jul 12 17:03:14.035484 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.035443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" event={"ID":"ef8e334d27c2be36757cd606a2fea1d6","Type":"ContainerStarted","Data":"e8fc1beda29d329feec67ac17f7053eb7bbd37c50cbde292d7b52f3aaa1ca3d5"} Jul 12 17:03:14.043508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-cni-bin\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.043611 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.043611 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043536 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmjv\" (UniqueName: \"kubernetes.io/projected/fb942b28-ce95-4401-bb85-b25474302d61-kube-api-access-lqmjv\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.043704 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-cni-bin\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.043739 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-etc-kubernetes\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.043771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.043803 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043779 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-etc-kubernetes\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.043837 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-run-netns\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.043837 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.043837 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043831 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.043990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00c5aa6e-c15c-4b8a-b201-44c4355ef991-serviceca\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.043990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043865 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-run-netns\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.043990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-daemon-config\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.043990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-registration-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.043990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-host\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-env-overrides\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-var-lib-kubelet\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b559k\" (UniqueName: \"kubernetes.io/projected/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-kube-api-access-b559k\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-etc-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb942b28-ce95-4401-bb85-b25474302d61-ovn-node-metrics-cert\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-modprobe-d\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-host-slash\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-slash\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00c5aa6e-c15c-4b8a-b201-44c4355ef991-serviceca\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-ovn\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-node-log\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-daemon-config\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-log-socket\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.043907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-socket-dir-parent\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-netns\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-kubernetes\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-iptables-alerter-script\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-registration-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044393 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-host\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-node-log\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00c5aa6e-c15c-4b8a-b201-44c4355ef991-host\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysctl-d\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-tmp-dir\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-systemd-units\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1811a7cf-f4d2-4424-a69d-aa8b9e402759-tmp\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.044686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-k8s-cni-cncf-io\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cnibin\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-sys-fs\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysconfig\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-etc-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-systemd\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-cni-bin\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8px44\" (UniqueName: \"kubernetes.io/projected/cf74a844-f099-4a72-804e-0106c3b44a3c-kube-api-access-8px44\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-host-slash\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-system-cni-dir\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-run\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044711 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-lib-modules\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-os-release\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-modprobe-d\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.045551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-systemd\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-systemd\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044811 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-env-overrides\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-conf-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-log-socket\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-ovn\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044854 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-conf-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9pt\" (UniqueName: \"kubernetes.io/projected/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-kube-api-access-db9pt\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-var-lib-kubelet\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqp6\" (UniqueName: \"kubernetes.io/projected/1811a7cf-f4d2-4424-a69d-aa8b9e402759-kube-api-access-2tqp6\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044924 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cnibin\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-sys-fs\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-var-lib-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-socket-dir-parent\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044901 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.044998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a35afdb-8438-4880-a499-ff342dee3802-agent-certs\") pod \"konnectivity-agent-4qprc\" (UID: \"8a35afdb-8438-4880-a499-ff342dee3802\") " pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-multus-certs\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-slash\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.046448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cni-binary-copy\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00c5aa6e-c15c-4b8a-b201-44c4355ef991-host\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-tuned\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a35afdb-8438-4880-a499-ff342dee3802-konnectivity-ca\") pod \"konnectivity-agent-4qprc\" (UID: \"8a35afdb-8438-4880-a499-ff342dee3802\") " pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-kubernetes\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf74a844-f099-4a72-804e-0106c3b44a3c-cni-binary-copy\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-cni-multus\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045220 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-cni-multus\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045220 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-cni-bin\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysctl-d\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447h2\" (UniqueName: \"kubernetes.io/projected/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-kube-api-access-447h2\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-netns\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045320 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-system-cni-dir\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-run\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-socket-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-var-lib-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045444 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-lib-modules\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.047440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysconfig\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-socket-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.045487 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045098 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-systemd-units\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.045566 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:14.545528043 +0000 UTC m=+2.040434468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-multus-certs\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-systemd\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.045943 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-run-k8s-cni-cncf-io\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-os-release\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046154 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046163 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046426 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-tmp-dir\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cni-binary-copy\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-iptables-alerter-script\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-hosts-file\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl89h\" (UniqueName: \"kubernetes.io/projected/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-kube-api-access-hl89h\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.048283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvm5k\" (UniqueName: \"kubernetes.io/projected/00c5aa6e-c15c-4b8a-b201-44c4355ef991-kube-api-access-tvm5k\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-kubelet\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-sys\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.046684 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a35afdb-8438-4880-a499-ff342dee3802-konnectivity-ca\") pod \"konnectivity-agent-4qprc\" (UID: \"8a35afdb-8438-4880-a499-ff342dee3802\") " pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-kubelet\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-hosts-file\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-kubelet\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.046749 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:14.546716978 +0000 UTC m=+2.041623398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046773 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-host-var-lib-kubelet\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046773 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-sys\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-ovnkube-config\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046845 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-run-openvswitch\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-cnibin\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-os-release\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.048887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-hostroot\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-cnibin\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.046988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef941278-2d82-4baa-bb36-caa777b09ab0-kubelet-config\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-os-release\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047028 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef941278-2d82-4baa-bb36-caa777b09ab0-dbus\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047048 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-hostroot\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-device-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef941278-2d82-4baa-bb36-caa777b09ab0-kubelet-config\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-ovnkube-script-lib\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zh9\" (UniqueName: \"kubernetes.io/projected/7115496d-2b24-4f09-b65a-678040e4a303-kube-api-access-99zh9\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-device-dir\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-system-cni-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysctl-conf\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-cni-netd\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047237 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef941278-2d82-4baa-bb36-caa777b09ab0-dbus\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-cni-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-multus-cni-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.049587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-ovnkube-config\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-sysctl-conf\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb942b28-ce95-4401-bb85-b25474302d61-host-cni-netd\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047808 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb942b28-ce95-4401-bb85-b25474302d61-ovnkube-script-lib\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf74a844-f099-4a72-804e-0106c3b44a3c-system-cni-dir\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.047942 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.048284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf74a844-f099-4a72-804e-0106c3b44a3c-cni-binary-copy\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.048497 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1811a7cf-f4d2-4424-a69d-aa8b9e402759-etc-tuned\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.048532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1811a7cf-f4d2-4424-a69d-aa8b9e402759-tmp\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.048650 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb942b28-ce95-4401-bb85-b25474302d61-ovn-node-metrics-cert\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.050122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.049593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a35afdb-8438-4880-a499-ff342dee3802-agent-certs\") pod \"konnectivity-agent-4qprc\" (UID: \"8a35afdb-8438-4880-a499-ff342dee3802\") " pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:14.052254 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.052236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmjv\" (UniqueName: \"kubernetes.io/projected/fb942b28-ce95-4401-bb85-b25474302d61-kube-api-access-lqmjv\") pod \"ovnkube-node-dvmx2\" (UID: \"fb942b28-ce95-4401-bb85-b25474302d61\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.052330 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.052241 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:14.052330 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.052286 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:14.052330 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.052297 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:14.052435 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.052357 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:14.552347551 +0000 UTC m=+2.047253973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:14.052888 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.052871 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b559k\" (UniqueName: \"kubernetes.io/projected/4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743-kube-api-access-b559k\") pod \"iptables-alerter-wt8w7\" (UID: \"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743\") " pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.054560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.054536 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8px44\" (UniqueName: \"kubernetes.io/projected/cf74a844-f099-4a72-804e-0106c3b44a3c-kube-api-access-8px44\") pod \"multus-58k2l\" (UID: \"cf74a844-f099-4a72-804e-0106c3b44a3c\") " pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.055633 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.055608 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvm5k\" (UniqueName: \"kubernetes.io/projected/00c5aa6e-c15c-4b8a-b201-44c4355ef991-kube-api-access-tvm5k\") pod \"node-ca-x48ln\" (UID: \"00c5aa6e-c15c-4b8a-b201-44c4355ef991\") " pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.056203 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.056179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl89h\" (UniqueName: \"kubernetes.io/projected/1223e5b6-e4d5-4c8d-bf67-e3723e46ea51-kube-api-access-hl89h\") pod \"node-resolver-8jk4d\" (UID: \"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51\") " pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.056373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.056355 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-447h2\" (UniqueName: \"kubernetes.io/projected/eab4f6c6-ddc2-4337-aa2b-e1f41e811818-kube-api-access-447h2\") pod \"multus-additional-cni-plugins-v79f6\" (UID: \"eab4f6c6-ddc2-4337-aa2b-e1f41e811818\") " pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.056440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.056399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqp6\" (UniqueName: \"kubernetes.io/projected/1811a7cf-f4d2-4424-a69d-aa8b9e402759-kube-api-access-2tqp6\") pod \"tuned-dsp6f\" (UID: \"1811a7cf-f4d2-4424-a69d-aa8b9e402759\") " pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.056981 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.056963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9pt\" (UniqueName: \"kubernetes.io/projected/fa8bfebe-ea5d-4de6-911b-eae14cd4a11a-kube-api-access-db9pt\") pod \"aws-ebs-csi-driver-node-s6kb6\" (UID: \"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.057064 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.056989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zh9\" (UniqueName: \"kubernetes.io/projected/7115496d-2b24-4f09-b65a-678040e4a303-kube-api-access-99zh9\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:14.165188 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.165162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58k2l" Jul 12 17:03:14.174561 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.174531 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf74a844_f099_4a72_804e_0106c3b44a3c.slice/crio-58e0b22a1dc624bdddba24dd1769e4103e5f39983517ed268a56d73be7f73aca WatchSource:0}: Error finding container 58e0b22a1dc624bdddba24dd1769e4103e5f39983517ed268a56d73be7f73aca: Status 404 returned error can't find the container with id 58e0b22a1dc624bdddba24dd1769e4103e5f39983517ed268a56d73be7f73aca Jul 12 17:03:14.177608 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.177591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v79f6" Jul 12 17:03:14.185330 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.185312 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab4f6c6_ddc2_4337_aa2b_e1f41e811818.slice/crio-90ed8d99122c8df27a2269873b820888de5a34497f731a4df7bcc8c64a4caebc WatchSource:0}: Error finding container 90ed8d99122c8df27a2269873b820888de5a34497f731a4df7bcc8c64a4caebc: Status 404 returned error can't find the container with id 90ed8d99122c8df27a2269873b820888de5a34497f731a4df7bcc8c64a4caebc Jul 12 17:03:14.195139 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.195120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" Jul 12 17:03:14.197257 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.197242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" Jul 12 17:03:14.205026 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.205007 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8bfebe_ea5d_4de6_911b_eae14cd4a11a.slice/crio-c32d0e97ce21d07b4585ee7d467096394a22d56bd3dd02fe1d5395fc68d2c751 WatchSource:0}: Error finding container c32d0e97ce21d07b4585ee7d467096394a22d56bd3dd02fe1d5395fc68d2c751: Status 404 returned error can't find the container with id c32d0e97ce21d07b4585ee7d467096394a22d56bd3dd02fe1d5395fc68d2c751 Jul 12 17:03:14.205721 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.205704 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1811a7cf_f4d2_4424_a69d_aa8b9e402759.slice/crio-f3e82b2302962dd5893d440c3e0a7c6e9d9e85a22d01936a00035426a28bceb2 WatchSource:0}: Error finding container f3e82b2302962dd5893d440c3e0a7c6e9d9e85a22d01936a00035426a28bceb2: Status 404 returned error can't find the container with id f3e82b2302962dd5893d440c3e0a7c6e9d9e85a22d01936a00035426a28bceb2 Jul 12 17:03:14.229753 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.229724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jk4d" Jul 12 17:03:14.236720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.236698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wt8w7" Jul 12 17:03:14.245267 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.245244 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc6a4fd_bcfd_4487_8f4f_6c1d6088b743.slice/crio-7bb0c17156731b7d59338ee30f69815efd299861da5de4fbb5cd3f4d1207c4c3 WatchSource:0}: Error finding container 7bb0c17156731b7d59338ee30f69815efd299861da5de4fbb5cd3f4d1207c4c3: Status 404 returned error can't find the container with id 7bb0c17156731b7d59338ee30f69815efd299861da5de4fbb5cd3f4d1207c4c3 Jul 12 17:03:14.253525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.253509 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:14.261996 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.261968 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb942b28_ce95_4401_bb85_b25474302d61.slice/crio-538e0369cab6548bd49933e9368a4fc4fdaef5401fc4722fdf7e69c6c5e66cee WatchSource:0}: Error finding container 538e0369cab6548bd49933e9368a4fc4fdaef5401fc4722fdf7e69c6c5e66cee: Status 404 returned error can't find the container with id 538e0369cab6548bd49933e9368a4fc4fdaef5401fc4722fdf7e69c6c5e66cee Jul 12 17:03:14.264939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.264922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:14.269858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.269843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x48ln" Jul 12 17:03:14.272630 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.272612 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a35afdb_8438_4880_a499_ff342dee3802.slice/crio-7f218c984a80e63b8543f6236b978f9da56ef2c31dda49e87d5039d2d1d14a79 WatchSource:0}: Error finding container 7f218c984a80e63b8543f6236b978f9da56ef2c31dda49e87d5039d2d1d14a79: Status 404 returned error can't find the container with id 7f218c984a80e63b8543f6236b978f9da56ef2c31dda49e87d5039d2d1d14a79 Jul 12 17:03:14.278662 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:14.278631 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c5aa6e_c15c_4b8a_b201_44c4355ef991.slice/crio-3ec16321e89857cbf5504a4008e130b4a4a2691e4ec04311f60ec36f186796d8 WatchSource:0}: Error finding container 3ec16321e89857cbf5504a4008e130b4a4a2691e4ec04311f60ec36f186796d8: Status 404 returned error can't find the container with id 3ec16321e89857cbf5504a4008e130b4a4a2691e4ec04311f60ec36f186796d8 Jul 12 17:03:14.432320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.432293 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:14.550970 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.550878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:14.550970 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.550950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:14.551180 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.551094 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:14.551180 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.551152 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:15.551133698 +0000 UTC m=+3.046040120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:14.551569 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.551548 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:14.551646 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.551606 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:15.551591478 +0000 UTC m=+3.046497904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:14.651991 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.651951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:14.652227 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.652108 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:14.652227 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.652126 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:14.652227 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.652138 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:14.652227 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:14.652192 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:15.652175098 +0000 UTC m=+3.147081533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:14.882421 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.882283 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 16:58:13 +0000 UTC" deadline="2028-03-19 11:13:54.061875506 +0000 UTC" Jul 12 17:03:14.882421 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:14.882337 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14778h10m39.179542743s" Jul 12 17:03:15.035012 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.034969 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:15.035191 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.035109 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:15.039252 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.038954 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:15.044794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.044760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x48ln" event={"ID":"00c5aa6e-c15c-4b8a-b201-44c4355ef991","Type":"ContainerStarted","Data":"3ec16321e89857cbf5504a4008e130b4a4a2691e4ec04311f60ec36f186796d8"} Jul 12 17:03:15.050205 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.050170 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4qprc" event={"ID":"8a35afdb-8438-4880-a499-ff342dee3802","Type":"ContainerStarted","Data":"7f218c984a80e63b8543f6236b978f9da56ef2c31dda49e87d5039d2d1d14a79"} Jul 12 17:03:15.065474 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.065373 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wt8w7" event={"ID":"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743","Type":"ContainerStarted","Data":"7bb0c17156731b7d59338ee30f69815efd299861da5de4fbb5cd3f4d1207c4c3"} Jul 12 17:03:15.084878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.084704 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jk4d" event={"ID":"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51","Type":"ContainerStarted","Data":"54f18a70e192f337e9bdc457097b430ecb882a89a0ecf0eb439f75fce0a1f56c"} Jul 12 17:03:15.091558 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.091504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" event={"ID":"1811a7cf-f4d2-4424-a69d-aa8b9e402759","Type":"ContainerStarted","Data":"f3e82b2302962dd5893d440c3e0a7c6e9d9e85a22d01936a00035426a28bceb2"} Jul 12 17:03:15.100735 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.100698 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerStarted","Data":"90ed8d99122c8df27a2269873b820888de5a34497f731a4df7bcc8c64a4caebc"} Jul 12 17:03:15.125030 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.124898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58k2l" event={"ID":"cf74a844-f099-4a72-804e-0106c3b44a3c","Type":"ContainerStarted","Data":"58e0b22a1dc624bdddba24dd1769e4103e5f39983517ed268a56d73be7f73aca"} Jul 12 17:03:15.142316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.142199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"538e0369cab6548bd49933e9368a4fc4fdaef5401fc4722fdf7e69c6c5e66cee"} Jul 12 17:03:15.150550 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.150463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" event={"ID":"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a","Type":"ContainerStarted","Data":"c32d0e97ce21d07b4585ee7d467096394a22d56bd3dd02fe1d5395fc68d2c751"} Jul 12 17:03:15.163698 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.163658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" event={"ID":"bafc608fa264eba1d850d33cd5b6db88","Type":"ContainerStarted","Data":"75c24e12f18c5a0a3c48b1b94e3d88bbd956c1765d9c2d2ffc47858e36546391"} Jul 12 17:03:15.559495 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.559405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:15.559654 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.559516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:15.559654 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.559555 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:15.559654 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.559628 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:17.559608101 +0000 UTC m=+5.054514532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:15.559654 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.559634 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:15.559881 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.559684 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:17.559669142 +0000 UTC m=+5.054575575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:15.660460 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.660420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:15.660650 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.660584 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:15.660650 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.660605 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:15.660650 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.660617 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:15.660803 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:15.660677 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:17.660658457 +0000 UTC m=+5.155564896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:15.807544 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.807510 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:15.882818 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.882727 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 16:58:13 +0000 UTC" deadline="2028-04-02 04:28:09.201284358 +0000 UTC" Jul 12 17:03:15.882818 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:15.882769 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15107h24m53.318521119s" Jul 12 17:03:16.033960 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:16.033199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:16.033960 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:16.033355 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:16.033960 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:16.033770 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:16.033960 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:16.033880 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:16.054956 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:16.054542 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 17:03:17.033643 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:17.033173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:17.033643 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.033295 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:17.575949 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:17.575896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:17.576142 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:17.576027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:17.576206 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.576162 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:17.576255 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.576225 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:21.576206202 +0000 UTC m=+9.071112625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:17.576627 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.576611 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:17.576709 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.576664 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:21.576647284 +0000 UTC m=+9.071553702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:17.677233 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:17.676591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:17.677233 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.676767 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:17.677233 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.676790 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:17.677233 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.676803 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:17.677233 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:17.676861 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:21.676842787 +0000 UTC m=+9.171749222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:18.033980 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:18.033231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:18.033980 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:18.033245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:18.033980 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:18.033362 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:18.033980 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:18.033491 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:19.033552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:19.033124 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:19.033552 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:19.033251 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:20.033261 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:20.032621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:20.033261 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:20.032746 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:20.033261 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:20.033116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:20.033261 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:20.033216 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:21.038594 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:21.038358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:21.038990 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.038690 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:21.607731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:21.607695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:21.607944 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:21.607771 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:21.607944 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.607876 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:21.607944 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.607928 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:21.608100 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.607952 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:29.607935003 +0000 UTC m=+17.102841428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:21.608100 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.607996 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:29.607976415 +0000 UTC m=+17.102882849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:21.708532 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:21.708490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:21.708715 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.708684 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:21.708715 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.708712 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:21.708827 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.708726 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:21.708827 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:21.708791 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:29.708772773 +0000 UTC m=+17.203679205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:22.032779 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:22.032747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:22.032779 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:22.032771 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:22.033017 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:22.032884 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:22.033097 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:22.033063 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:23.032529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:23.032494 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:23.032997 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:23.032627 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:24.032577 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:24.032491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:24.032577 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:24.032548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:24.033102 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:24.032639 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:24.033102 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:24.032799 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:25.035876 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:25.035845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:25.036318 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:25.035973 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:26.033264 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:26.033230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:26.033398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:26.033242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:26.033398 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:26.033342 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:26.033506 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:26.033411 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:27.033002 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:27.032963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:27.033456 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:27.033102 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:28.033601 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:28.033565 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:28.034042 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:28.033574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:28.034042 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:28.033678 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:28.034042 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:28.033791 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:29.033123 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:29.033082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:29.033303 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.033223 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:29.676348 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:29.676310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:29.676348 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:29.676356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:29.676828 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.676454 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:29.676828 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.676518 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:45.676498428 +0000 UTC m=+33.171404866 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:29.676828 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.676521 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:29.676828 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.676570 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:45.676555345 +0000 UTC m=+33.171461782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:29.776629 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:29.776589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:29.776778 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.776736 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:29.776778 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.776754 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:29.776778 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.776765 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:29.776968 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:29.776826 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:45.776809277 +0000 UTC m=+33.271715696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:30.033595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:30.033559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:30.033783 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:30.033614 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:30.033783 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:30.033687 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:30.033878 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:30.033815 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:31.032898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:31.032862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:31.033290 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:31.033018 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:32.033203 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:32.033162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:32.033635 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:32.033286 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:32.033635 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:32.033324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:32.033635 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:32.033404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:33.036763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:33.036731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:33.037193 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:33.036848 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:34.033519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:34.033484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:34.033723 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:34.033609 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:34.033723 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:34.033672 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:34.033838 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:34.033799 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:35.033699 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.033399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:35.034367 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:35.033793 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:35.207724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.207688 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x48ln" event={"ID":"00c5aa6e-c15c-4b8a-b201-44c4355ef991","Type":"ContainerStarted","Data":"47b81bbd401ac959ee87a80e50805112a025680a05c6cf290ef2bdb6c3e20a53"} Jul 12 17:03:35.209304 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.209270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4qprc" event={"ID":"8a35afdb-8438-4880-a499-ff342dee3802","Type":"ContainerStarted","Data":"e8c2a05ef95314b39bd08373298cdfde7310ae200fa17e1b53f0da40b2cdb677"} Jul 12 17:03:35.210771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.210743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jk4d" event={"ID":"1223e5b6-e4d5-4c8d-bf67-e3723e46ea51","Type":"ContainerStarted","Data":"bb4aefc04f8e55aedb5b44c476ea77a6f2f32b1ff8b0dfa019d66621fe29fe9d"} Jul 12 17:03:35.212605 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.212574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" event={"ID":"1811a7cf-f4d2-4424-a69d-aa8b9e402759","Type":"ContainerStarted","Data":"e4f870f911e7dd5f73e499a4340d268eff73966e1df52c3b187aab23b1e35a56"} Jul 12 17:03:35.214306 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.214284 2573 generic.go:356] "Generic (PLEG): container finished" podID="eab4f6c6-ddc2-4337-aa2b-e1f41e811818" containerID="8884460a9473f9cde652e517a9aaef49a79951913ccfc6c4490c066ac3c630b3" exitCode=0 Jul 12 17:03:35.214405 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.214367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerDied","Data":"8884460a9473f9cde652e517a9aaef49a79951913ccfc6c4490c066ac3c630b3"} Jul 12 17:03:35.216004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.215980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58k2l" event={"ID":"cf74a844-f099-4a72-804e-0106c3b44a3c","Type":"ContainerStarted","Data":"567042ad77f0b4c5a21e1a7be6814e8203de4609a982affe319a2766165a1456"} Jul 12 17:03:35.217601 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.217566 2573 generic.go:356] "Generic (PLEG): container finished" podID="ef8e334d27c2be36757cd606a2fea1d6" containerID="b0528866d848229e20132534de9035f4475679f6ea40d0e670d76534ba32f983" exitCode=0 Jul 12 17:03:35.217601 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.217593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" event={"ID":"ef8e334d27c2be36757cd606a2fea1d6","Type":"ContainerDied","Data":"b0528866d848229e20132534de9035f4475679f6ea40d0e670d76534ba32f983"} Jul 12 17:03:35.220279 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220257 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:03:35.220672 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220645 2573 generic.go:356] "Generic (PLEG): container finished" podID="fb942b28-ce95-4401-bb85-b25474302d61" containerID="ca66d3e6462338cdce99c6d88616282db246cf6ab7b44fe09a1f99ccf0fea530" exitCode=1 Jul 12 17:03:35.220771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220719 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"dbf87d85277c97e1140bf41da761807b27ecdc2985937db9a2e424e3ffdbd2ce"} Jul 12 17:03:35.220834 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"faea91d5edf0cff1ae36de6dee58a58375c0cd8f745b8e8641488d0605af0d7f"} Jul 12 17:03:35.220892 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220847 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"64af50c8efa88b25f3d19ba909c9dd93fae14167c79f59e241f8b26a32c72c8c"} Jul 12 17:03:35.220892 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"9da0267265f1d6eed023fcbc20641653848fb838fdbc35be9ea2e0c5353b2e58"} Jul 12 17:03:35.220999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerDied","Data":"ca66d3e6462338cdce99c6d88616282db246cf6ab7b44fe09a1f99ccf0fea530"} Jul 12 17:03:35.220999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.220978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"ecd7b1300d229c422b3d3ad45f392ff65705795f1c2574b47c99134beb9018d9"} Jul 12 17:03:35.222187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.222168 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" event={"ID":"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a","Type":"ContainerStarted","Data":"30798e742926efadf0ea4128b2d84292202db547c92151c68dc882891dd88a5a"} Jul 12 17:03:35.223410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.223392 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" event={"ID":"bafc608fa264eba1d850d33cd5b6db88","Type":"ContainerStarted","Data":"6c9ae81e41063f7ad036b9dad09f3238aad588276e985f5aa0590c3f92f9eac1"} Jul 12 17:03:35.228337 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.228298 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x48ln" podStartSLOduration=2.269259023 podStartE2EDuration="22.228286181s" totalImagesPullingTime="19.959027158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.227530523 +0000 UTC m=+22.722436965" watchObservedRunningTime="2026-07-12 17:03:35.228286181 +0000 UTC m=+22.723192661" Jul 12 17:03:35.242975 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.242943 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jk4d" podStartSLOduration=2.245059311 podStartE2EDuration="22.24293153s" totalImagesPullingTime="19.997872219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.24258075 +0000 UTC m=+22.737487189" watchObservedRunningTime="2026-07-12 17:03:35.24293153 +0000 UTC m=+22.737837969" Jul 12 17:03:35.261015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.260955 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-89.ec2.internal" podStartSLOduration=22.260936707 podStartE2EDuration="22.260936707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.260589613 +0000 UTC m=+22.755496054" watchObservedRunningTime="2026-07-12 17:03:35.260936707 +0000 UTC m=+22.755843146" Jul 12 17:03:35.320877 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.320837 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dsp6f" podStartSLOduration=2.25814113 podStartE2EDuration="22.320821847s" totalImagesPullingTime="20.062680717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.32045973 +0000 UTC m=+22.815366170" watchObservedRunningTime="2026-07-12 17:03:35.320821847 +0000 UTC m=+22.815728287" Jul 12 17:03:35.337861 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.337824 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4qprc" podStartSLOduration=7.012445995 podStartE2EDuration="22.337808165s" totalImagesPullingTime="15.32536217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.337394657 +0000 UTC m=+22.832301096" watchObservedRunningTime="2026-07-12 17:03:35.337808165 +0000 UTC m=+22.832714605" Jul 12 17:03:35.358756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.358712 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-58k2l" podStartSLOduration=1.999998697 podStartE2EDuration="22.358697772s" totalImagesPullingTime="20.358699075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.358458606 +0000 UTC m=+22.853365046" watchObservedRunningTime="2026-07-12 17:03:35.358697772 +0000 UTC m=+22.853604208" Jul 12 17:03:35.479238 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:35.479199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:36.009629 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.009601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:36.032659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.032633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:36.032787 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:36.032732 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:36.032854 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.032787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:36.032949 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:36.032927 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:36.114133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.114111 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 17:03:36.231066 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.230989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" event={"ID":"ef8e334d27c2be36757cd606a2fea1d6","Type":"ContainerStarted","Data":"c3b1514493c35643a866072afac485bccb65ba00f923b420eeaeac96ca824366"} Jul 12 17:03:36.233047 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.233014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" event={"ID":"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a","Type":"ContainerStarted","Data":"99c8c35c31136951b4abb67b74324c98fb90238e5bb0b26e1c3cce66cb5f2ab8"} Jul 12 17:03:36.234584 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.234531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wt8w7" event={"ID":"4bc6a4fd-bcfd-4487-8f4f-6c1d6088b743","Type":"ContainerStarted","Data":"f26ba125afe577de04640a9396e5ef6e7b9b6dff2391d2ac31b76407f175fbda"} Jul 12 17:03:36.246404 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.246356 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-89.ec2.internal" podStartSLOduration=23.246344414 podStartE2EDuration="23.246344414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:36.24564424 +0000 UTC m=+23.740550681" watchObservedRunningTime="2026-07-12 17:03:36.246344414 +0000 UTC m=+23.741250854" Jul 12 17:03:36.259862 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.259813 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wt8w7" podStartSLOduration=3.23666024 podStartE2EDuration="23.259796397s" totalImagesPullingTime="20.023136157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:36.259304112 +0000 UTC m=+23.754210552" watchObservedRunningTime="2026-07-12 17:03:36.259796397 +0000 UTC m=+23.754702837" Jul 12 17:03:36.266504 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.266460 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:36.676870 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.676836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:36.781284 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.781241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:36.917935 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.917819 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T17:03:36.114127181Z","UUID":"b30f6d28-4a1f-4772-a521-31374b3fcf05","Handler":null,"Name":"","Endpoint":""} Jul 12 17:03:36.919772 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.919744 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 17:03:36.919952 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.919788 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 17:03:36.922309 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:36.922283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:37.021017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:37.020905 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:37.032944 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:37.032906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:37.033066 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:37.033020 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:37.123470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:37.123433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:37.261969 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:37.261941 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:37.262591 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:37.262572 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:37.482482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:37.482293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:38.032972 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.032938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:38.033299 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:38.033055 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:38.033299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.033110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:38.033299 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:38.033243 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:38.241839 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.241810 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:03:38.242369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.242177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"c3207484fb81b89616c52f00baa19f7a75eaab50331f69568046eadb97442e2b"} Jul 12 17:03:38.244418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.244390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" event={"ID":"fa8bfebe-ea5d-4de6-911b-eae14cd4a11a","Type":"ContainerStarted","Data":"a2bc81478bc3d17ccbcff5231f2e6a361ae8ee237b49246e65908fc248117d4d"} Jul 12 17:03:38.244967 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.244728 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:38.245112 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.245097 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4qprc" Jul 12 17:03:38.265979 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.265935 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6kb6" podStartSLOduration=2.878278865 podStartE2EDuration="25.265904942s" totalImagesPullingTime="22.387626077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:38.265378856 +0000 UTC m=+25.760285299" watchObservedRunningTime="2026-07-12 17:03:38.265904942 +0000 UTC m=+25.760811386" Jul 12 17:03:38.277412 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.277378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:38.505150 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.505119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:38.866422 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:38.866347 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:39.033431 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:39.033400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:39.033613 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:39.033519 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:39.070402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:39.070371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:39.194227 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:39.194191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:39.680344 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:39.680195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:40.033098 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.033066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:40.033288 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.033066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:40.033288 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:40.033167 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:40.033288 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:40.033251 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:40.251404 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.251373 2573 generic.go:356] "Generic (PLEG): container finished" podID="eab4f6c6-ddc2-4337-aa2b-e1f41e811818" containerID="eb6d06d8b234a10bdffb270b587631486566a0980ff00be4a4c156fc2c7ba117" exitCode=0 Jul 12 17:03:40.251587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.251455 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerDied","Data":"eb6d06d8b234a10bdffb270b587631486566a0980ff00be4a4c156fc2c7ba117"} Jul 12 17:03:40.254797 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.254778 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:03:40.255131 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.255108 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"79a14d99b956d9829271e33627ed34f1606d5cbbc815ac7b8c4f5896f418a017"} Jul 12 17:03:40.255451 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.255367 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:40.255451 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.255423 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:40.255642 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.255471 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:40.255642 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.255545 2573 scope.go:117] "RemoveContainer" containerID="ca66d3e6462338cdce99c6d88616282db246cf6ab7b44fe09a1f99ccf0fea530" Jul 12 17:03:40.272263 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.272244 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:40.272972 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.272954 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:03:40.273474 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.273458 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:40.475471 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.475444 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:40.887611 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:40.887581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:41.018256 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.018221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:41.033292 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.033271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:41.033375 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:41.033360 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:41.118149 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.118122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:41.261689 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.261670 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:03:41.262011 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.261990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" event={"ID":"fb942b28-ce95-4401-bb85-b25474302d61","Type":"ContainerStarted","Data":"4f65543cb6173a4d6532552326c7551e89875f9889435f12baf6e2530e8f7fc9"} Jul 12 17:03:41.299714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.299670 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" podStartSLOduration=8.078431802 podStartE2EDuration="28.299655222s" totalImagesPullingTime="20.22122342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:41.297439402 +0000 UTC m=+28.792345843" watchObservedRunningTime="2026-07-12 17:03:41.299655222 +0000 UTC m=+28.794561664" Jul 12 17:03:41.475236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:41.475204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:42.033088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.033049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:42.033555 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:42.033176 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:42.033555 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.033220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:42.033555 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:42.033320 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:42.266173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.266141 2573 generic.go:356] "Generic (PLEG): container finished" podID="eab4f6c6-ddc2-4337-aa2b-e1f41e811818" containerID="16d342ce2a8e7702b7b30f358399effd5ea130861509c6dd32f7b0e5b4ba2ebc" exitCode=0 Jul 12 17:03:42.266343 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.266231 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerDied","Data":"16d342ce2a8e7702b7b30f358399effd5ea130861509c6dd32f7b0e5b4ba2ebc"} Jul 12 17:03:42.267438 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.267305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:42.474541 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.474509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:42.585823 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.585791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:42.685501 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.685466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:42.882343 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:42.882264 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:43.033423 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:43.033394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:43.033786 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:43.033481 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:44.032728 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:44.032550 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:44.032900 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:44.032550 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:44.032900 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:44.032815 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:44.032900 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:44.032882 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:44.272239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:44.272205 2573 generic.go:356] "Generic (PLEG): container finished" podID="eab4f6c6-ddc2-4337-aa2b-e1f41e811818" containerID="1f3f9860339b4e5893d980f1ed82b10e398c32c2505bf7098beb4afa7d6a1a79" exitCode=0 Jul 12 17:03:44.272610 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:44.272261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerDied","Data":"1f3f9860339b4e5893d980f1ed82b10e398c32c2505bf7098beb4afa7d6a1a79"} Jul 12 17:03:45.032957 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:45.032904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:45.033143 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.033061 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:45.695264 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:45.695230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:45.695713 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:45.695301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:45.695713 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.695401 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:45.695713 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.695444 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:45.695713 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.695465 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret podName:ef941278-2d82-4baa-bb36-caa777b09ab0 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:17.695447248 +0000 UTC m=+65.190353667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret") pod "global-pull-secret-syncer-tw7w7" (UID: "ef941278-2d82-4baa-bb36-caa777b09ab0") : object "kube-system"/"original-pull-secret" not registered Jul 12 17:03:45.695713 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.695504 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs podName:7115496d-2b24-4f09-b65a-678040e4a303 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:17.695487277 +0000 UTC m=+65.190393700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs") pod "network-metrics-daemon-clc6z" (UID: "7115496d-2b24-4f09-b65a-678040e4a303") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 17:03:45.796033 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:45.795993 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:45.796235 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.796216 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 17:03:45.796283 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.796243 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 17:03:45.796283 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.796256 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nw5gj for pod openshift-network-diagnostics/network-check-target-zjsj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:45.796368 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:45.796317 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj podName:5e8a2231-b475-49ed-83e9-e2ace6150f34 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:17.796298242 +0000 UTC m=+65.291204666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nw5gj" (UniqueName: "kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj") pod "network-check-target-zjsj4" (UID: "5e8a2231-b475-49ed-83e9-e2ace6150f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 17:03:46.030584 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:46.030448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:46.033321 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:46.033297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:46.033454 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:46.033406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:46.033535 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:46.033482 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:46.033617 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:46.033578 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:46.134355 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:46.134319 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:47.033414 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:47.033380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:47.033881 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:47.033518 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:48.033277 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:48.033239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:48.033465 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:48.033239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:48.033465 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:48.033364 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:48.033838 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:48.033476 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:49.033066 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:49.033022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:49.033269 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:49.033155 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:49.286324 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:49.286244 2573 generic.go:356] "Generic (PLEG): container finished" podID="eab4f6c6-ddc2-4337-aa2b-e1f41e811818" containerID="81a33bdfc9186df778b234474e506c35a64eea6da6d04a4ebf935182f81f11ca" exitCode=0 Jul 12 17:03:49.286324 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:49.286305 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerDied","Data":"81a33bdfc9186df778b234474e506c35a64eea6da6d04a4ebf935182f81f11ca"} Jul 12 17:03:50.033443 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.033403 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:50.033651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.033404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:50.033651 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:50.033505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:50.033651 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:50.033615 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:50.210335 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.210298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:50.294057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.293959 2573 generic.go:356] "Generic (PLEG): container finished" podID="eab4f6c6-ddc2-4337-aa2b-e1f41e811818" containerID="e1c23a6d3eec13da71eb9122eaded8c543414dbf10fe18681614d0e1f9c76db8" exitCode=0 Jul 12 17:03:50.294057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.294023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerDied","Data":"e1c23a6d3eec13da71eb9122eaded8c543414dbf10fe18681614d0e1f9c76db8"} Jul 12 17:03:50.308050 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.308012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:50.461354 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.461322 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:50.691161 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.691125 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:50.792862 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:50.792833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.032855 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.032762 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:51.033006 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:51.032868 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:51.058980 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.058948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.214509 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.214478 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.299389 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.299301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v79f6" event={"ID":"eab4f6c6-ddc2-4337-aa2b-e1f41e811818","Type":"ContainerStarted","Data":"975cba8ed92b789933ebba05ee7d1ba973abbe52ce6eb90c62977fb6651d441b"} Jul 12 17:03:51.328990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.328947 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v79f6" podStartSLOduration=7.612218111 podStartE2EDuration="38.328931047s" totalImagesPullingTime="30.716712936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:51.328146115 +0000 UTC m=+38.823052556" watchObservedRunningTime="2026-07-12 17:03:51.328931047 +0000 UTC m=+38.823837508" Jul 12 17:03:51.349127 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.349102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.509206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.509172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.620237 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.620159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.715798 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.715765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:51.813850 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:51.813811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:52.033055 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.033019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:52.033055 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.033055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:52.033258 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:52.033153 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:52.033323 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:52.033296 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:52.097522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.097491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:52.245451 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.245417 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:52.486445 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.486417 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:52.507686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.507609 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zjsj4"] Jul 12 17:03:52.507874 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.507747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:52.507955 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:52.507874 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:52.508571 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.508548 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tw7w7"] Jul 12 17:03:52.508692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.508656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:52.508784 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:52.508746 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:52.509202 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.509183 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-clc6z"] Jul 12 17:03:52.509306 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:52.509275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:52.509375 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:52.509358 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:54.033751 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:54.033461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:54.033751 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:54.033541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:54.034197 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:54.033765 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:54.034197 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:54.033548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:54.034197 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:54.033831 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:54.034197 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:54.033897 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:55.254933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:55.254874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.353805 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:55.353769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.428497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:55.428465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.499015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:55.498982 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.032807 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.032764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:56.032807 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.032801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:56.032990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.032814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:56.032990 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:56.032898 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zjsj4" podUID="5e8a2231-b475-49ed-83e9-e2ace6150f34" Jul 12 17:03:56.033054 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:56.033013 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tw7w7" podUID="ef941278-2d82-4baa-bb36-caa777b09ab0" Jul 12 17:03:56.033099 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:03:56.033080 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clc6z" podUID="7115496d-2b24-4f09-b65a-678040e4a303" Jul 12 17:03:56.035851 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.035793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.092924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.092881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.285883 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.285804 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-89.ec2.internal" event="NodeReady" Jul 12 17:03:56.286259 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.285936 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 17:03:56.320503 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.320468 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9cc8fc59b-t2mh4"] Jul 12 17:03:56.350201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.350171 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9cc8fc59b-t2mh4"] Jul 12 17:03:56.350201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.350206 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wd2sd"] Jul 12 17:03:56.350434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.350314 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.353508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.353482 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 12 17:03:56.353645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.353623 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xrlpt\"" Jul 12 17:03:56.353714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.353635 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 12 17:03:56.353979 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.353962 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 12 17:03:56.359596 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.359575 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 12 17:03:56.364552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.364522 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wctm5"] Jul 12 17:03:56.364552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.364552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.367351 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.367335 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvzd7\"" Jul 12 17:03:56.367470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.367335 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 17:03:56.367470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.367335 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 17:03:56.386529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.386498 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wctm5"] Jul 12 17:03:56.386529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.386535 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wd2sd"] Jul 12 17:03:56.386700 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.386631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.389326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.389304 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 17:03:56.389326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.389316 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 17:03:56.389506 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.389385 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 17:03:56.389506 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.389418 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vs9jl\"" Jul 12 17:03:56.389506 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.389457 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 17:03:56.439212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.439177 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-df2gd"] Jul 12 17:03:56.454587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.454556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.458958 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.458941 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 17:03:56.459123 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.459106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xwq56\"" Jul 12 17:03:56.459211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.459199 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 17:03:56.459723 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.459706 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 17:03:56.464135 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.464114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-df2gd"] Jul 12 17:03:56.471651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b9bcb2-7f37-4544-a569-eca348e5e03b-config-volume\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.471758 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9kb\" (UniqueName: \"kubernetes.io/projected/a2b9bcb2-7f37-4544-a569-eca348e5e03b-kube-api-access-kd9kb\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.471758 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-registry-tls\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471758 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-trusted-ca\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471758 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-installation-pull-secrets\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471758 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-image-registry-private-configuration\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471934 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b9bcb2-7f37-4544-a569-eca348e5e03b-metrics-tls\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.471934 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-ca-trust-extracted\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471934 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjd4t\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-kube-api-access-cjd4t\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471934 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-bound-sa-token\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471934 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-registry-certificates\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.471934 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.471903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b9bcb2-7f37-4544-a569-eca348e5e03b-tmp-dir\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.530124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.530087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.573261 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27e62b21-0b33-483d-8806-430a8ebb21c5-crio-socket\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.573261 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcgj\" (UniqueName: \"kubernetes.io/projected/27e62b21-0b33-483d-8806-430a8ebb21c5-kube-api-access-mzcgj\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.573261 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-bound-sa-token\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.573519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-registry-certificates\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.573519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w42\" (UniqueName: \"kubernetes.io/projected/1f22e4c8-ca08-4c16-80f2-c31d9eb24d70-kube-api-access-m6w42\") pod \"ingress-canary-df2gd\" (UID: \"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70\") " pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.573519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27e62b21-0b33-483d-8806-430a8ebb21c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.573519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b9bcb2-7f37-4544-a569-eca348e5e03b-tmp-dir\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.573519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b9bcb2-7f37-4544-a569-eca348e5e03b-config-volume\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.573519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9kb\" (UniqueName: \"kubernetes.io/projected/a2b9bcb2-7f37-4544-a569-eca348e5e03b-kube-api-access-kd9kb\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.573799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-registry-tls\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.573799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27e62b21-0b33-483d-8806-430a8ebb21c5-data-volume\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.573799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27e62b21-0b33-483d-8806-430a8ebb21c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.573799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-trusted-ca\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.573799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b9bcb2-7f37-4544-a569-eca348e5e03b-tmp-dir\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.573799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573773 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-installation-pull-secrets\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.574081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f22e4c8-ca08-4c16-80f2-c31d9eb24d70-cert\") pod \"ingress-canary-df2gd\" (UID: \"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70\") " pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.574081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-image-registry-private-configuration\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.574081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b9bcb2-7f37-4544-a569-eca348e5e03b-metrics-tls\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.574081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.573968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-ca-trust-extracted\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.574081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.574000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjd4t\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-kube-api-access-cjd4t\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.574322 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.574080 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b9bcb2-7f37-4544-a569-eca348e5e03b-config-volume\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.574322 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.574222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-registry-certificates\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.574426 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.574388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-ca-trust-extracted\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.574982 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.574958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-trusted-ca\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.577997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.577975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-registry-tls\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.577997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.577986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b9bcb2-7f37-4544-a569-eca348e5e03b-metrics-tls\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.577997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.577995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-installation-pull-secrets\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.578166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.578073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-image-registry-private-configuration\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.581626 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.581593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9kb\" (UniqueName: \"kubernetes.io/projected/a2b9bcb2-7f37-4544-a569-eca348e5e03b-kube-api-access-kd9kb\") pod \"dns-default-wd2sd\" (UID: \"a2b9bcb2-7f37-4544-a569-eca348e5e03b\") " pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.582049 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.582032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjd4t\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-kube-api-access-cjd4t\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.582101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.582032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db122e2-3b4a-4d3f-b533-eb1ce590b4b3-bound-sa-token\") pod \"image-registry-9cc8fc59b-t2mh4\" (UID: \"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3\") " pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.595556 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.595525 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.660504 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.660473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:56.673633 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.673605 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wd2sd" Jul 12 17:03:56.674847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.674826 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27e62b21-0b33-483d-8806-430a8ebb21c5-crio-socket\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.674926 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.674855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcgj\" (UniqueName: \"kubernetes.io/projected/27e62b21-0b33-483d-8806-430a8ebb21c5-kube-api-access-mzcgj\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.674926 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.674878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w42\" (UniqueName: \"kubernetes.io/projected/1f22e4c8-ca08-4c16-80f2-c31d9eb24d70-kube-api-access-m6w42\") pod \"ingress-canary-df2gd\" (UID: \"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70\") " pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.675022 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.674903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27e62b21-0b33-483d-8806-430a8ebb21c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.675022 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.674962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27e62b21-0b33-483d-8806-430a8ebb21c5-data-volume\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.675022 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.674988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27e62b21-0b33-483d-8806-430a8ebb21c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.675149 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.675029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f22e4c8-ca08-4c16-80f2-c31d9eb24d70-cert\") pod \"ingress-canary-df2gd\" (UID: \"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70\") " pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.675149 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.675077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27e62b21-0b33-483d-8806-430a8ebb21c5-crio-socket\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.675413 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.675383 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27e62b21-0b33-483d-8806-430a8ebb21c5-data-volume\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.675566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.675550 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27e62b21-0b33-483d-8806-430a8ebb21c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.677245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.677221 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27e62b21-0b33-483d-8806-430a8ebb21c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.677333 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.677305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f22e4c8-ca08-4c16-80f2-c31d9eb24d70-cert\") pod \"ingress-canary-df2gd\" (UID: \"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70\") " pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.683196 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.683167 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcgj\" (UniqueName: \"kubernetes.io/projected/27e62b21-0b33-483d-8806-430a8ebb21c5-kube-api-access-mzcgj\") pod \"insights-runtime-extractor-wctm5\" (UID: \"27e62b21-0b33-483d-8806-430a8ebb21c5\") " pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.683744 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.683723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w42\" (UniqueName: \"kubernetes.io/projected/1f22e4c8-ca08-4c16-80f2-c31d9eb24d70-kube-api-access-m6w42\") pod \"ingress-canary-df2gd\" (UID: \"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70\") " pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.695304 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.695281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wctm5" Jul 12 17:03:56.763176 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.763142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-df2gd" Jul 12 17:03:56.889637 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.889567 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9cc8fc59b-t2mh4"] Jul 12 17:03:56.890256 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.890231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wd2sd"] Jul 12 17:03:56.895107 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:56.895057 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db122e2_3b4a_4d3f_b533_eb1ce590b4b3.slice/crio-c702ff171f2178add007b482e982630fcbb4e9af045dc665e502abe7964c3f88 WatchSource:0}: Error finding container c702ff171f2178add007b482e982630fcbb4e9af045dc665e502abe7964c3f88: Status 404 returned error can't find the container with id c702ff171f2178add007b482e982630fcbb4e9af045dc665e502abe7964c3f88 Jul 12 17:03:56.897354 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:56.896646 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b9bcb2_7f37_4544_a569_eca348e5e03b.slice/crio-017b66b8501678c9fc3eaaa316043404993ead11d0859af8f657d122f036f15c WatchSource:0}: Error finding container 017b66b8501678c9fc3eaaa316043404993ead11d0859af8f657d122f036f15c: Status 404 returned error can't find the container with id 017b66b8501678c9fc3eaaa316043404993ead11d0859af8f657d122f036f15c Jul 12 17:03:56.914141 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.914111 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wctm5"] Jul 12 17:03:56.917595 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:56.917569 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e62b21_0b33_483d_8806_430a8ebb21c5.slice/crio-305ed18068cb90a24e2d7c96bd792d7a3aae79823df5a3a38db2e8d3be5f3896 WatchSource:0}: Error finding container 305ed18068cb90a24e2d7c96bd792d7a3aae79823df5a3a38db2e8d3be5f3896: Status 404 returned error can't find the container with id 305ed18068cb90a24e2d7c96bd792d7a3aae79823df5a3a38db2e8d3be5f3896 Jul 12 17:03:56.949086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:56.948932 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-df2gd"] Jul 12 17:03:56.951895 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:03:56.951866 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f22e4c8_ca08_4c16_80f2_c31d9eb24d70.slice/crio-730615ee7c4fbc9ed7ed90c6615547fd46dbb11687005ceed678fb7a4f852792 WatchSource:0}: Error finding container 730615ee7c4fbc9ed7ed90c6615547fd46dbb11687005ceed678fb7a4f852792: Status 404 returned error can't find the container with id 730615ee7c4fbc9ed7ed90c6615547fd46dbb11687005ceed678fb7a4f852792 Jul 12 17:03:57.318655 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.318616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wd2sd" event={"ID":"a2b9bcb2-7f37-4544-a569-eca348e5e03b","Type":"ContainerStarted","Data":"017b66b8501678c9fc3eaaa316043404993ead11d0859af8f657d122f036f15c"} Jul 12 17:03:57.320351 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.320319 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-df2gd" event={"ID":"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70","Type":"ContainerStarted","Data":"730615ee7c4fbc9ed7ed90c6615547fd46dbb11687005ceed678fb7a4f852792"} Jul 12 17:03:57.323299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.323271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" event={"ID":"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3","Type":"ContainerStarted","Data":"0b19bbd740ad1a8a251eccfa9df8269b303e41cc46807dea8dfa6b52c7572c70"} Jul 12 17:03:57.323430 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.323317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" event={"ID":"9db122e2-3b4a-4d3f-b533-eb1ce590b4b3","Type":"ContainerStarted","Data":"c702ff171f2178add007b482e982630fcbb4e9af045dc665e502abe7964c3f88"} Jul 12 17:03:57.323488 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.323451 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:03:57.325065 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.325004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wctm5" event={"ID":"27e62b21-0b33-483d-8806-430a8ebb21c5","Type":"ContainerStarted","Data":"305ed18068cb90a24e2d7c96bd792d7a3aae79823df5a3a38db2e8d3be5f3896"} Jul 12 17:03:57.338973 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.338934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:57.345350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.345283 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" podStartSLOduration=6.345263243 podStartE2EDuration="6.345263243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:57.344726525 +0000 UTC m=+44.839632995" watchObservedRunningTime="2026-07-12 17:03:57.345263243 +0000 UTC m=+44.840169676" Jul 12 17:03:57.396172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:57.396113 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:58.032940 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.032700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:03:58.033303 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.033008 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:03:58.033936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.033477 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:03:58.042682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.037982 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 17:03:58.042682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.038307 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 17:03:58.042682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.038630 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgjd4\"" Jul 12 17:03:58.042682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.038902 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 17:03:58.042682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.040280 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 17:03:58.042682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.040500 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wpssr\"" Jul 12 17:03:58.461963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.461869 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:58.549609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.549576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:58.922045 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:03:58.922006 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:00.968684 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:00.968634 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:01.027090 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:01.027053 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:03.064152 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.064119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:03.177451 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.177421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:03.342740 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.342516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wd2sd" event={"ID":"a2b9bcb2-7f37-4544-a569-eca348e5e03b","Type":"ContainerStarted","Data":"7029d581944af72eb129f178001e7138f8147e2e10209e96183085a5805817da"} Jul 12 17:04:03.342936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.342776 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wd2sd" Jul 12 17:04:03.342936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.342806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wd2sd" event={"ID":"a2b9bcb2-7f37-4544-a569-eca348e5e03b","Type":"ContainerStarted","Data":"f8afd0280c813402737072015e185e86ffecb38dc1f0389700b79ec47cda4465"} Jul 12 17:04:03.343890 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.343867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-df2gd" event={"ID":"1f22e4c8-ca08-4c16-80f2-c31d9eb24d70","Type":"ContainerStarted","Data":"9b294e240bd441d0628bb7e6e8cd2b802d8c8db13063482e07b950b43e1a9c17"} Jul 12 17:04:03.345278 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.345258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wctm5" event={"ID":"27e62b21-0b33-483d-8806-430a8ebb21c5","Type":"ContainerStarted","Data":"1e269f1846f84f2c1d1184dab3a24117c9b1ad7c5ae6ff0d314a1b08c90f24ec"} Jul 12 17:04:03.360774 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.360720 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wd2sd" podStartSLOduration=1.337514947 podStartE2EDuration="7.360702157s" totalImagesPullingTime="6.02318721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:03.358957472 +0000 UTC m=+50.853863913" watchObservedRunningTime="2026-07-12 17:04:03.360702157 +0000 UTC m=+50.855608596" Jul 12 17:04:03.376514 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.376449 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-df2gd" podStartSLOduration=1.4008756390000001 podStartE2EDuration="7.376429202s" totalImagesPullingTime="5.975553563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:03.375295901 +0000 UTC m=+50.870202343" watchObservedRunningTime="2026-07-12 17:04:03.376429202 +0000 UTC m=+50.871335643" Jul 12 17:04:03.461183 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.461071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:03.523325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:03.523278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:04.350186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:04.350155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wctm5" event={"ID":"27e62b21-0b33-483d-8806-430a8ebb21c5","Type":"ContainerStarted","Data":"a68e7af7229c9f377bcb5fe42f61fbac275b393b94fc3791e6eb49c751bbec1c"} Jul 12 17:04:06.412975 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:06.412906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:07.362727 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:07.362693 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wctm5" event={"ID":"27e62b21-0b33-483d-8806-430a8ebb21c5","Type":"ContainerStarted","Data":"695bfa1bf5bdc4b2eb6e3c998ac43d69f25a4f5bd48fb25664fa565d3bf51157"} Jul 12 17:04:07.383325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:07.383274 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wctm5" podStartSLOduration=1.6678677309999999 podStartE2EDuration="11.383255683s" totalImagesPullingTime="9.715387952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:07.381517703 +0000 UTC m=+54.876424143" watchObservedRunningTime="2026-07-12 17:04:07.383255683 +0000 UTC m=+54.878162168" Jul 12 17:04:07.656062 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:07.656034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:07.764035 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:07.764001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:08.064876 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.064796 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79fd67c46-z2qsl"] Jul 12 17:04:08.087132 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.087099 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79fd67c46-z2qsl"] Jul 12 17:04:08.087275 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.087239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.090996 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.090953 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 12 17:04:08.091511 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.090953 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 12 17:04:08.091638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.091554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 12 17:04:08.091638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.091505 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 12 17:04:08.092084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.092058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2nhhq\"" Jul 12 17:04:08.092239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.092206 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 12 17:04:08.095749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.093037 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 12 17:04:08.095749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.093225 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 12 17:04:08.105522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.105495 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 12 17:04:08.171893 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.171864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-oauth-config\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.172092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.171959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-oauth-serving-cert\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.172092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.171982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-trusted-ca-bundle\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.172092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.172000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zm6\" (UniqueName: \"kubernetes.io/projected/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-kube-api-access-b9zm6\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.172092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.172038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-serving-cert\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.172253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.172120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-config\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.172253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.172148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-service-ca\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273237 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-oauth-serving-cert\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273237 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-trusted-ca-bundle\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273462 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zm6\" (UniqueName: \"kubernetes.io/projected/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-kube-api-access-b9zm6\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273462 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-serving-cert\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-config\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273617 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273558 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-service-ca\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.273617 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.273595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-oauth-config\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.274203 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.274161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-oauth-serving-cert\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.274343 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.274314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-config\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.274415 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.274341 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-service-ca\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.274683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.274657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-trusted-ca-bundle\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.277625 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.277606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-oauth-config\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.277698 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.277658 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-serving-cert\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.281796 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.281766 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zm6\" (UniqueName: \"kubernetes.io/projected/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-kube-api-access-b9zm6\") pod \"console-79fd67c46-z2qsl\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.405954 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.405903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:08.562983 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.562727 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79fd67c46-z2qsl"] Jul 12 17:04:08.567596 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:08.567551 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4ed7b0_6a56_4d9f_a691_a111575e1c61.slice/crio-b318f1511bd8f0924724ea39d0565bf2f0188ac7290baef4e39ee94678ace4fa WatchSource:0}: Error finding container b318f1511bd8f0924724ea39d0565bf2f0188ac7290baef4e39ee94678ace4fa: Status 404 returned error can't find the container with id b318f1511bd8f0924724ea39d0565bf2f0188ac7290baef4e39ee94678ace4fa Jul 12 17:04:08.622547 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.622514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:08.853443 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.853352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:08.912794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.912764 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-49k5k"] Jul 12 17:04:08.934587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.934550 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pbw2r"] Jul 12 17:04:08.934738 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.934693 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:08.939220 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.939194 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 17:04:08.939539 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.939514 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 12 17:04:08.939674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.939523 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 12 17:04:08.939674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.939567 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 17:04:08.939794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.939734 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 17:04:08.940448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.940424 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w46f6\"" Jul 12 17:04:08.940568 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.940530 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 12 17:04:08.948617 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.948584 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-49k5k"] Jul 12 17:04:08.948761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.948707 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.951253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.951218 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 17:04:08.951253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.951254 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 17:04:08.951434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.951220 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 17:04:08.951434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.951371 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbqq8\"" Jul 12 17:04:08.960796 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.960763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:08.980389 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksvv\" (UniqueName: \"kubernetes.io/projected/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-api-access-zksvv\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:08.980580 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980580 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-tls\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980580 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bafc58cf-1e74-4ce0-8907-252eebde85c6-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:08.980580 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:08.980756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:08.980756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bafc58cf-1e74-4ce0-8907-252eebde85c6-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:08.980756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdrr\" (UniqueName: \"kubernetes.io/projected/3d84b57e-568b-4ce9-b514-5b348444bc66-kube-api-access-vvdrr\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-sys\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-root\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d84b57e-568b-4ce9-b514-5b348444bc66-metrics-client-ca\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-wtmp\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-textfile\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:08.980984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:08.980900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.050821 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.050790 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.081898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.081864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-tls\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.081902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bafc58cf-1e74-4ce0-8907-252eebde85c6-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.082082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.081955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.082082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.081986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.082082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bafc58cf-1e74-4ce0-8907-252eebde85c6-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.082082 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:04:09.082031 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 12 17:04:09.082082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdrr\" (UniqueName: \"kubernetes.io/projected/3d84b57e-568b-4ce9-b514-5b348444bc66-kube-api-access-vvdrr\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-sys\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:04:09.082129 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-tls podName:3d84b57e-568b-4ce9-b514-5b348444bc66 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:09.582103067 +0000 UTC m=+57.077009493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-tls") pod "node-exporter-pbw2r" (UID: "3d84b57e-568b-4ce9-b514-5b348444bc66") : secret "node-exporter-tls" not found Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-sys\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-root\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-root\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d84b57e-568b-4ce9-b514-5b348444bc66-metrics-client-ca\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-wtmp\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:04:09.082131 2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-textfile\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:04:09.082398 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-tls podName:bafc58cf-1e74-4ce0-8907-252eebde85c6 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:09.582379335 +0000 UTC m=+57.077285768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-49k5k" (UID: "bafc58cf-1e74-4ce0-8907-252eebde85c6") : secret "kube-state-metrics-tls" not found Jul 12 17:04:09.082402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bafc58cf-1e74-4ce0-8907-252eebde85c6-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksvv\" (UniqueName: \"kubernetes.io/projected/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-api-access-zksvv\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082752 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-textfile\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082813 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082879 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-wtmp\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d84b57e-568b-4ce9-b514-5b348444bc66-metrics-client-ca\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.082932 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bafc58cf-1e74-4ce0-8907-252eebde85c6-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.083084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.083055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.086359 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.086333 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.086482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.086431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.093378 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.093350 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdrr\" (UniqueName: \"kubernetes.io/projected/3d84b57e-568b-4ce9-b514-5b348444bc66-kube-api-access-vvdrr\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.098201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.098175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksvv\" (UniqueName: \"kubernetes.io/projected/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-api-access-zksvv\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.155111 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.155079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.269646 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.269610 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.359356 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.359318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.369775 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.369745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fd67c46-z2qsl" event={"ID":"7d4ed7b0-6a56-4d9f-a691-a111575e1c61","Type":"ContainerStarted","Data":"b318f1511bd8f0924724ea39d0565bf2f0188ac7290baef4e39ee94678ace4fa"} Jul 12 17:04:09.447231 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.447148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.587143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.587109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-tls\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.587143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.587157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.589731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.589704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d84b57e-568b-4ce9-b514-5b348444bc66-node-exporter-tls\") pod \"node-exporter-pbw2r\" (UID: \"3d84b57e-568b-4ce9-b514-5b348444bc66\") " pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.589855 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.589804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bafc58cf-1e74-4ce0-8907-252eebde85c6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-49k5k\" (UID: \"bafc58cf-1e74-4ce0-8907-252eebde85c6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.610689 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.610640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.706858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.706772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.798663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.798632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.846045 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.846014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" Jul 12 17:04:09.858238 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.858164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbw2r" Jul 12 17:04:09.901429 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:09.901386 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:09.909445 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:09.909419 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d84b57e_568b_4ce9_b514_5b348444bc66.slice/crio-f87092a74b59ab2d3bd1462c684b1775233c4fccc633f3995bce8dbca9e5658f WatchSource:0}: Error finding container f87092a74b59ab2d3bd1462c684b1775233c4fccc633f3995bce8dbca9e5658f: Status 404 returned error can't find the container with id f87092a74b59ab2d3bd1462c684b1775233c4fccc633f3995bce8dbca9e5658f Jul 12 17:04:10.024330 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:10.024297 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:10.045071 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:10.045037 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-49k5k"] Jul 12 17:04:10.049809 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:10.049779 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafc58cf_1e74_4ce0_8907_252eebde85c6.slice/crio-fa2707b8ab4b6c0f6fbd7a35db880eaacb7d5c527e413d8b3034bc3a8511a9ad WatchSource:0}: Error finding container fa2707b8ab4b6c0f6fbd7a35db880eaacb7d5c527e413d8b3034bc3a8511a9ad: Status 404 returned error can't find the container with id fa2707b8ab4b6c0f6fbd7a35db880eaacb7d5c527e413d8b3034bc3a8511a9ad Jul 12 17:04:10.150098 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:10.150066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:10.375141 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:10.375041 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbw2r" event={"ID":"3d84b57e-568b-4ce9-b514-5b348444bc66","Type":"ContainerStarted","Data":"f87092a74b59ab2d3bd1462c684b1775233c4fccc633f3995bce8dbca9e5658f"} Jul 12 17:04:10.376365 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:10.376340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" event={"ID":"bafc58cf-1e74-4ce0-8907-252eebde85c6","Type":"ContainerStarted","Data":"fa2707b8ab4b6c0f6fbd7a35db880eaacb7d5c527e413d8b3034bc3a8511a9ad"} Jul 12 17:04:11.052806 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.052775 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:11.383318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.383282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fd67c46-z2qsl" event={"ID":"7d4ed7b0-6a56-4d9f-a691-a111575e1c61","Type":"ContainerStarted","Data":"6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8"} Jul 12 17:04:11.407215 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.406255 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79fd67c46-z2qsl" podStartSLOduration=0.709735902 podStartE2EDuration="3.406231699s" totalImagesPullingTime="2.696495797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:11.403856586 +0000 UTC m=+58.898763029" watchObservedRunningTime="2026-07-12 17:04:11.406231699 +0000 UTC m=+58.901138142" Jul 12 17:04:11.490424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.490388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:11.878762 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.878701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:11.987896 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.987865 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55f764f5c4-rsj95"] Jul 12 17:04:11.990192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.990165 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:11.991541 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.991519 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:11.998433 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.998408 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 12 17:04:11.999060 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.999036 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 12 17:04:11.999172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.999137 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4q0d3r9dpkdep\"" Jul 12 17:04:11.999396 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.999366 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 12 17:04:11.999510 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.999367 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 12 17:04:11.999574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.999543 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vnfzm\"" Jul 12 17:04:11.999628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:11.999572 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 12 17:04:12.007236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.007210 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55f764f5c4-rsj95"] Jul 12 17:04:12.113287 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-tls\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d009e2f0-2268-4d72-b83f-8a53484b8b81-metrics-client-ca\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-grpc-tls\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9f4h\" (UniqueName: \"kubernetes.io/projected/d009e2f0-2268-4d72-b83f-8a53484b8b81-kube-api-access-x9f4h\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.113620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.113610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.144447 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.144419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:12.215053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-grpc-tls\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9f4h\" (UniqueName: \"kubernetes.io/projected/d009e2f0-2268-4d72-b83f-8a53484b8b81-kube-api-access-x9f4h\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-tls\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d009e2f0-2268-4d72-b83f-8a53484b8b81-metrics-client-ca\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.215339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.215311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.216202 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.216106 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d009e2f0-2268-4d72-b83f-8a53484b8b81-metrics-client-ca\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.218338 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.218302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.218545 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.218522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-grpc-tls\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.218756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.218730 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.219856 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.219833 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.220134 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.220111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-tls\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.220408 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.220386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d009e2f0-2268-4d72-b83f-8a53484b8b81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.223419 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.223397 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9f4h\" (UniqueName: \"kubernetes.io/projected/d009e2f0-2268-4d72-b83f-8a53484b8b81-kube-api-access-x9f4h\") pod \"thanos-querier-55f764f5c4-rsj95\" (UID: \"d009e2f0-2268-4d72-b83f-8a53484b8b81\") " pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.280492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.280370 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvmx2" Jul 12 17:04:12.302368 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.302340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:12.388830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.388760 2573 generic.go:356] "Generic (PLEG): container finished" podID="3d84b57e-568b-4ce9-b514-5b348444bc66" containerID="4abbee299fb7e4933ee60e16d24a8ba375fe770a32a34d5c4db40e438ad001c9" exitCode=0 Jul 12 17:04:12.388974 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.388844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbw2r" event={"ID":"3d84b57e-568b-4ce9-b514-5b348444bc66","Type":"ContainerDied","Data":"4abbee299fb7e4933ee60e16d24a8ba375fe770a32a34d5c4db40e438ad001c9"} Jul 12 17:04:12.394391 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.393896 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" event={"ID":"bafc58cf-1e74-4ce0-8907-252eebde85c6","Type":"ContainerStarted","Data":"c88b2b0cce105738db3ede2c1bdd4e9d82f6478bce599500ad23151caaf2bd45"} Jul 12 17:04:12.394391 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.393997 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" event={"ID":"bafc58cf-1e74-4ce0-8907-252eebde85c6","Type":"ContainerStarted","Data":"43b6c1ce664a71af3b22bd5a17f3818b8e97d5ed1eda6ed64caafcedc1f5a56a"} Jul 12 17:04:12.394391 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.394029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" event={"ID":"bafc58cf-1e74-4ce0-8907-252eebde85c6","Type":"ContainerStarted","Data":"0899d914581cdb43d2724d969ba091e784c0ada4ed351fd7fe8bc3d6a886b4f8"} Jul 12 17:04:12.463874 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.463814 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-49k5k" podStartSLOduration=2.399085002 podStartE2EDuration="4.463798758s" totalImagesPullingTime="2.064713756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:12.462072478 +0000 UTC m=+59.956978935" watchObservedRunningTime="2026-07-12 17:04:12.463798758 +0000 UTC m=+59.958705197" Jul 12 17:04:12.469110 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.469076 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55f764f5c4-rsj95"] Jul 12 17:04:12.472679 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:12.472644 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd009e2f0_2268_4d72_b83f_8a53484b8b81.slice/crio-fa2d90ec4346d7310129b639d5bdc8f014b8527b78550b4046f8256feee713f5 WatchSource:0}: Error finding container fa2d90ec4346d7310129b639d5bdc8f014b8527b78550b4046f8256feee713f5: Status 404 returned error can't find the container with id fa2d90ec4346d7310129b639d5bdc8f014b8527b78550b4046f8256feee713f5 Jul 12 17:04:12.490987 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.490943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:12.587295 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.587251 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:12.674869 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.674838 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:12.772385 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.772340 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:12.872999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.872972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:12.974352 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:12.974262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:13.353098 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.353018 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wd2sd" Jul 12 17:04:13.401733 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.401685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbw2r" event={"ID":"3d84b57e-568b-4ce9-b514-5b348444bc66","Type":"ContainerStarted","Data":"ab410b410d3b0a72df6df628048afc04b03e6950d930a49bc2d2b08bb2cdf68a"} Jul 12 17:04:13.401924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.401784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbw2r" event={"ID":"3d84b57e-568b-4ce9-b514-5b348444bc66","Type":"ContainerStarted","Data":"a3c765af3ddc104e5b1d64d27f212c6a0df998ce5bef0f1edd3f98b410dc69cd"} Jul 12 17:04:13.403092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.403009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"fa2d90ec4346d7310129b639d5bdc8f014b8527b78550b4046f8256feee713f5"} Jul 12 17:04:13.424332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.424273 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pbw2r" podStartSLOduration=3.142401643 podStartE2EDuration="5.424250987s" totalImagesPullingTime="2.202170691s" totalInitContainerRuntime="79.678653ms" isStatefulPod=true podCreationTimestamp="2026-07-12 17:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:13.422421413 +0000 UTC m=+60.917327854" watchObservedRunningTime="2026-07-12 17:04:13.424250987 +0000 UTC m=+60.919157427" Jul 12 17:04:13.689416 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.689380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:13.750587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.750547 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79fd67c46-z2qsl"] Jul 12 17:04:13.837778 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:13.837745 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:14.069643 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.069566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:14.111101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.111078 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq"] Jul 12 17:04:14.115471 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.115454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.118896 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.118875 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sx5d8\"" Jul 12 17:04:14.118896 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.118896 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 12 17:04:14.119044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.118964 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 12 17:04:14.120128 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.120108 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 12 17:04:14.120232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.120151 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 12 17:04:14.120394 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.120374 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 12 17:04:14.125569 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.125549 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 12 17:04:14.128014 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.127990 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq"] Jul 12 17:04:14.238059 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-secret-telemeter-client\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-federate-client-tls\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-telemeter-client-tls\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-serving-certs-ca-bundle\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjx5\" (UniqueName: \"kubernetes.io/projected/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-kube-api-access-vsjx5\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.238398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.238291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-metrics-client-ca\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339233 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-telemeter-client-tls\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339233 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-serving-certs-ca-bundle\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339233 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjx5\" (UniqueName: \"kubernetes.io/projected/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-kube-api-access-vsjx5\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339514 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339514 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-metrics-client-ca\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339619 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-secret-telemeter-client\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339675 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.339730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.339680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-federate-client-tls\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.340083 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.340046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-serving-certs-ca-bundle\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.340318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.340294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.340410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.340301 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-metrics-client-ca\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.341838 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.341815 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-telemeter-client-tls\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.342112 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.342080 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.342416 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.342399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-federate-client-tls\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.342461 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.342444 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-secret-telemeter-client\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.352621 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.352596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjx5\" (UniqueName: \"kubernetes.io/projected/cefdbe10-39f7-4b6c-8c60-e9230740c6bc-kube-api-access-vsjx5\") pod \"telemeter-client-7fbbfd5d5c-qj7hq\" (UID: \"cefdbe10-39f7-4b6c-8c60-e9230740c6bc\") " pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.409173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.409138 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"ecaf1133bbdeabd112531c025cc5ad89da424a54389b02cb29773bca4fe19b98"} Jul 12 17:04:14.409639 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.409229 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"5bbae4ee8e4767b2a6d8189e92bc4564b6e43b32edc11f0a0c280eabbdb258ae"} Jul 12 17:04:14.409639 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.409256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"aa7a546540aea13276ad0127299e78ad22abe5fd9fe70781a88298df23d32dee"} Jul 12 17:04:14.427655 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.427627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" Jul 12 17:04:14.557056 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:14.557031 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq"] Jul 12 17:04:14.559327 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:14.559303 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefdbe10_39f7_4b6c_8c60_e9230740c6bc.slice/crio-814b5818d8b67b4f44bd7eeb42367ceb6b4b06332f184d53a8661cee37db6426 WatchSource:0}: Error finding container 814b5818d8b67b4f44bd7eeb42367ceb6b4b06332f184d53a8661cee37db6426: Status 404 returned error can't find the container with id 814b5818d8b67b4f44bd7eeb42367ceb6b4b06332f184d53a8661cee37db6426 Jul 12 17:04:15.163324 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.163293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:15.271698 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.271656 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:15.341358 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.341327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:15.415925 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.415834 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"85c24e25b02f05d0f0198cf476efebb0b0d29c074e175338fa4f05098ee33549"} Jul 12 17:04:15.415925 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.415903 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"ccfb95f925925c483358b8693e514fc86e1680a4e050e12604950b2f3f032611"} Jul 12 17:04:15.416362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.415967 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" event={"ID":"d009e2f0-2268-4d72-b83f-8a53484b8b81","Type":"ContainerStarted","Data":"75f28dfac6f39b7b74bf3f7cc46cb3a18b7e0829d15c0b735a9dd44d1cab53de"} Jul 12 17:04:15.416362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.416000 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:15.417021 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.417002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" event={"ID":"cefdbe10-39f7-4b6c-8c60-e9230740c6bc","Type":"ContainerStarted","Data":"814b5818d8b67b4f44bd7eeb42367ceb6b4b06332f184d53a8661cee37db6426"} Jul 12 17:04:15.446807 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:15.446749 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" podStartSLOduration=2.229847608 podStartE2EDuration="4.446732227s" totalImagesPullingTime="2.216884619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:15.444394512 +0000 UTC m=+62.939300953" watchObservedRunningTime="2026-07-12 17:04:15.446732227 +0000 UTC m=+62.941638680" Jul 12 17:04:16.027886 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:16.027853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:16.086209 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:16.086180 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:17.426708 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.426670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" event={"ID":"cefdbe10-39f7-4b6c-8c60-e9230740c6bc","Type":"ContainerStarted","Data":"3ce9eb73826eaf12ec1575116ab1ca9a057bb3fee903f07dd501c54214373ccc"} Jul 12 17:04:17.772742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.772709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:04:17.772742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.772748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:04:17.775316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.775293 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 17:04:17.776300 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.776286 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 17:04:17.785602 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.785578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115496d-2b24-4f09-b65a-678040e4a303-metrics-certs\") pod \"network-metrics-daemon-clc6z\" (UID: \"7115496d-2b24-4f09-b65a-678040e4a303\") " pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:04:17.785701 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.785583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef941278-2d82-4baa-bb36-caa777b09ab0-original-pull-secret\") pod \"global-pull-secret-syncer-tw7w7\" (UID: \"ef941278-2d82-4baa-bb36-caa777b09ab0\") " pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:04:17.854197 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.854170 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wpssr\"" Jul 12 17:04:17.862453 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.862435 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clc6z" Jul 12 17:04:17.870491 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.870467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tw7w7" Jul 12 17:04:17.874061 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.874029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:04:17.876838 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.876810 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 17:04:17.887720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.887694 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 17:04:17.898608 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:17.898576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5gj\" (UniqueName: \"kubernetes.io/projected/5e8a2231-b475-49ed-83e9-e2ace6150f34-kube-api-access-nw5gj\") pod \"network-check-target-zjsj4\" (UID: \"5e8a2231-b475-49ed-83e9-e2ace6150f34\") " pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:04:18.001747 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.001651 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-clc6z"] Jul 12 17:04:18.005103 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:18.005079 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7115496d_2b24_4f09_b65a_678040e4a303.slice/crio-39032a88c9b7d397c8c8723b47ee51b9e99399abc10faec71e8397875b76c298 WatchSource:0}: Error finding container 39032a88c9b7d397c8c8723b47ee51b9e99399abc10faec71e8397875b76c298: Status 404 returned error can't find the container with id 39032a88c9b7d397c8c8723b47ee51b9e99399abc10faec71e8397875b76c298 Jul 12 17:04:18.016360 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.016339 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tw7w7"] Jul 12 17:04:18.018828 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:18.018806 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef941278_2d82_4baa_bb36_caa777b09ab0.slice/crio-8ca0377744b9721eabdc03632115fc8f3b5c85dfb9de7958a37b24c05f0fa278 WatchSource:0}: Error finding container 8ca0377744b9721eabdc03632115fc8f3b5c85dfb9de7958a37b24c05f0fa278: Status 404 returned error can't find the container with id 8ca0377744b9721eabdc03632115fc8f3b5c85dfb9de7958a37b24c05f0fa278 Jul 12 17:04:18.164878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.164846 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgjd4\"" Jul 12 17:04:18.173305 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.173282 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:04:18.297864 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.297827 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zjsj4"] Jul 12 17:04:18.300601 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:18.300575 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8a2231_b475_49ed_83e9_e2ace6150f34.slice/crio-2958c26a855e5ed4ad300cb02e6a294b75c12ded135e646aaa35616d6aa14373 WatchSource:0}: Error finding container 2958c26a855e5ed4ad300cb02e6a294b75c12ded135e646aaa35616d6aa14373: Status 404 returned error can't find the container with id 2958c26a855e5ed4ad300cb02e6a294b75c12ded135e646aaa35616d6aa14373 Jul 12 17:04:18.332095 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.332074 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9cc8fc59b-t2mh4" Jul 12 17:04:18.406923 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.406870 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:18.433067 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.433036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zjsj4" event={"ID":"5e8a2231-b475-49ed-83e9-e2ace6150f34","Type":"ContainerStarted","Data":"2958c26a855e5ed4ad300cb02e6a294b75c12ded135e646aaa35616d6aa14373"} Jul 12 17:04:18.436079 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.436043 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" event={"ID":"cefdbe10-39f7-4b6c-8c60-e9230740c6bc","Type":"ContainerStarted","Data":"5e1c75c94767bedfb2ec164bab285dc133f666934d6868a811fd79e17602fc9f"} Jul 12 17:04:18.436213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.436113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" event={"ID":"cefdbe10-39f7-4b6c-8c60-e9230740c6bc","Type":"ContainerStarted","Data":"a654b2b1a73bb97a0fd5a568d791ee84374f5ceb393f7a26984f8fcb356c900b"} Jul 12 17:04:18.437939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.437877 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tw7w7" event={"ID":"ef941278-2d82-4baa-bb36-caa777b09ab0","Type":"ContainerStarted","Data":"8ca0377744b9721eabdc03632115fc8f3b5c85dfb9de7958a37b24c05f0fa278"} Jul 12 17:04:18.439451 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.439421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clc6z" event={"ID":"7115496d-2b24-4f09-b65a-678040e4a303","Type":"ContainerStarted","Data":"39032a88c9b7d397c8c8723b47ee51b9e99399abc10faec71e8397875b76c298"} Jul 12 17:04:18.459765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.459717 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fbbfd5d5c-qj7hq" podStartSLOduration=1.469961509 podStartE2EDuration="4.459699418s" totalImagesPullingTime="2.989737909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:18.457903625 +0000 UTC m=+65.952810102" watchObservedRunningTime="2026-07-12 17:04:18.459699418 +0000 UTC m=+65.954605858" Jul 12 17:04:18.462683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.462655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:18.561787 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:18.561694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:19.069875 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.069746 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:19.167306 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.167273 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:19.191593 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.191517 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54dcc7b7d9-jz2fw"] Jul 12 17:04:19.194973 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.194901 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.211035 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.210116 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dcc7b7d9-jz2fw"] Jul 12 17:04:19.287209 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmcn\" (UniqueName: \"kubernetes.io/projected/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-kube-api-access-cbmcn\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.287209 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-oauth-config\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.287434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-serving-cert\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.287434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-config\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.287434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-trusted-ca-bundle\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.287551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-oauth-serving-cert\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.287551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.287494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-service-ca\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.388587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-trusted-ca-bundle\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.388745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-oauth-serving-cert\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.388745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-service-ca\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.388745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmcn\" (UniqueName: \"kubernetes.io/projected/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-kube-api-access-cbmcn\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.388745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-oauth-config\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.389003 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-serving-cert\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.389003 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.388775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-config\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.389552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.389499 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-trusted-ca-bundle\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.389552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.389523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-oauth-serving-cert\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.389745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.389578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-config\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.389828 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.389799 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-service-ca\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.392178 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.392124 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-serving-cert\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.392281 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.392217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-oauth-config\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.398748 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.398703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmcn\" (UniqueName: \"kubernetes.io/projected/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-kube-api-access-cbmcn\") pod \"console-54dcc7b7d9-jz2fw\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.447266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.447173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clc6z" event={"ID":"7115496d-2b24-4f09-b65a-678040e4a303","Type":"ContainerStarted","Data":"3088b4a167b83ba40aa479cc0989c2e15b8353fb9df730a7fc1201720e3b663d"} Jul 12 17:04:19.447266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.447243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clc6z" event={"ID":"7115496d-2b24-4f09-b65a-678040e4a303","Type":"ContainerStarted","Data":"91b7442dd834572ef8c6d562e591fa3455b293b34bd962ea8594a2ab9a594591"} Jul 12 17:04:19.465423 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.464448 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-clc6z" podStartSLOduration=65.526643329 podStartE2EDuration="1m6.46443201s" totalImagesPullingTime="937.788681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:19.464233068 +0000 UTC m=+66.959139532" watchObservedRunningTime="2026-07-12 17:04:19.46443201 +0000 UTC m=+66.959338451" Jul 12 17:04:19.505893 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.505667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:19.574291 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.574145 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:19.637726 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.637664 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:19.688205 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.688171 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dcc7b7d9-jz2fw"] Jul 12 17:04:19.691845 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:04:19.691817 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f7c7a8_d7ec_4530_89cc_9efdf9d226b2.slice/crio-b6a0c1e3a5bf3a38fb6fe7aea2d38b26d274903feb90a41760691073cb7c6a20 WatchSource:0}: Error finding container b6a0c1e3a5bf3a38fb6fe7aea2d38b26d274903feb90a41760691073cb7c6a20: Status 404 returned error can't find the container with id b6a0c1e3a5bf3a38fb6fe7aea2d38b26d274903feb90a41760691073cb7c6a20 Jul 12 17:04:19.937280 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:19.937225 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:20.004787 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:20.004705 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:20.454218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:20.454185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dcc7b7d9-jz2fw" event={"ID":"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2","Type":"ContainerStarted","Data":"d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d"} Jul 12 17:04:20.454716 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:20.454242 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dcc7b7d9-jz2fw" event={"ID":"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2","Type":"ContainerStarted","Data":"b6a0c1e3a5bf3a38fb6fe7aea2d38b26d274903feb90a41760691073cb7c6a20"} Jul 12 17:04:20.477024 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:20.476966 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54dcc7b7d9-jz2fw" podStartSLOduration=1.47694539 podStartE2EDuration="1.47694539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:20.4764208 +0000 UTC m=+67.971327241" watchObservedRunningTime="2026-07-12 17:04:20.47694539 +0000 UTC m=+67.971851831" Jul 12 17:04:21.429671 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:21.429642 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55f764f5c4-rsj95" Jul 12 17:04:22.484059 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:22.484021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:22.553829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:22.553771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:23.126189 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.126160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:23.189724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.189698 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:23.467191 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.467149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tw7w7" event={"ID":"ef941278-2d82-4baa-bb36-caa777b09ab0","Type":"ContainerStarted","Data":"03326188f0ad0822513b1b36beea1e37e22420978c0c03f723a2113e4bbc2471"} Jul 12 17:04:23.468748 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.468724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zjsj4" event={"ID":"5e8a2231-b475-49ed-83e9-e2ace6150f34","Type":"ContainerStarted","Data":"9b2d32c99b65cdfcc9f3fcb59b8339cf2ea8a31f54321639a3fa10d898bf43c2"} Jul 12 17:04:23.468873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.468769 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:04:23.483384 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.483334 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tw7w7" podStartSLOduration=65.406560854 podStartE2EDuration="1m10.483314928s" totalImagesPullingTime="5.076754074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:23.48208925 +0000 UTC m=+70.976995690" watchObservedRunningTime="2026-07-12 17:04:23.483314928 +0000 UTC m=+70.978221370" Jul 12 17:04:23.499047 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:23.498999 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zjsj4" podStartSLOduration=65.714757153 podStartE2EDuration="1m10.498987172s" totalImagesPullingTime="4.784230019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:23.498411711 +0000 UTC m=+70.993318151" watchObservedRunningTime="2026-07-12 17:04:23.498987172 +0000 UTC m=+70.993893613" Jul 12 17:04:27.065312 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:27.065271 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:27.173318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:27.173288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:27.237411 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:27.237383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:29.505806 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:29.505775 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:29.506245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:29.505830 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:29.510522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:29.510499 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:30.496562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:30.496535 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:04:30.666010 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:30.665952 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:30.767802 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:30.767730 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:30.832879 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:30.832854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:31.027469 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:31.027399 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:31.088245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:31.088215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:31.502173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:31.502144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:31.574339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:31.574303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:36.302294 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:36.302174 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:38.772616 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:38.772569 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79fd67c46-z2qsl" podUID="7d4ed7b0-6a56-4d9f-a691-a111575e1c61" containerName="console" containerID="cri-o://6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8" gracePeriod=15 Jul 12 17:04:39.009995 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.009967 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:39.057435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057352 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-trusted-ca-bundle\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057400 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-config\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057426 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-service-ca\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057704 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057447 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-serving-cert\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057704 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057472 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-oauth-config\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057704 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057493 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9zm6\" (UniqueName: \"kubernetes.io/projected/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-kube-api-access-b9zm6\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057704 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057521 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-oauth-serving-cert\") pod \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\" (UID: \"7d4ed7b0-6a56-4d9f-a691-a111575e1c61\") " Jul 12 17:04:39.057904 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057845 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:04:39.058022 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.057995 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-service-ca" (OuterVolumeSpecName: "service-ca") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:04:39.058085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.058015 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-config" (OuterVolumeSpecName: "console-config") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:04:39.058157 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.058133 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:04:39.059900 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.059870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:04:39.060045 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.059970 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-kube-api-access-b9zm6" (OuterVolumeSpecName: "kube-api-access-b9zm6") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "kube-api-access-b9zm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:04:39.060045 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.059990 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7d4ed7b0-6a56-4d9f-a691-a111575e1c61" (UID: "7d4ed7b0-6a56-4d9f-a691-a111575e1c61"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:04:39.158053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158017 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.158053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158046 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-serving-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.158053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158056 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-oauth-config\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.158271 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158065 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zm6\" (UniqueName: \"kubernetes.io/projected/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-kube-api-access-b9zm6\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.158271 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158075 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-oauth-serving-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.158271 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158083 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-trusted-ca-bundle\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.158271 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.158090 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4ed7b0-6a56-4d9f-a691-a111575e1c61-console-config\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:04:39.523218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.523174 2573 generic.go:356] "Generic (PLEG): container finished" podID="7d4ed7b0-6a56-4d9f-a691-a111575e1c61" containerID="6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8" exitCode=0 Jul 12 17:04:39.523363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.523227 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fd67c46-z2qsl" Jul 12 17:04:39.523363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.523260 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fd67c46-z2qsl" event={"ID":"7d4ed7b0-6a56-4d9f-a691-a111575e1c61","Type":"ContainerDied","Data":"6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8"} Jul 12 17:04:39.523363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.523313 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fd67c46-z2qsl" event={"ID":"7d4ed7b0-6a56-4d9f-a691-a111575e1c61","Type":"ContainerDied","Data":"b318f1511bd8f0924724ea39d0565bf2f0188ac7290baef4e39ee94678ace4fa"} Jul 12 17:04:39.523363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.523340 2573 scope.go:117] "RemoveContainer" containerID="6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8" Jul 12 17:04:39.532127 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.532108 2573 scope.go:117] "RemoveContainer" containerID="6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8" Jul 12 17:04:39.532393 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:04:39.532373 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8\": container with ID starting with 6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8 not found: ID does not exist" containerID="6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8" Jul 12 17:04:39.532443 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.532401 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8"} err="failed to get container status \"6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8\": rpc error: code = NotFound desc = could not find container \"6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8\": container with ID starting with 6af10e403f1c993afb49a03b470009b3fe739538600d613bc0660e9916b44ec8 not found: ID does not exist" Jul 12 17:04:39.543312 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.543292 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79fd67c46-z2qsl"] Jul 12 17:04:39.547378 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:39.547360 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79fd67c46-z2qsl"] Jul 12 17:04:41.037540 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:41.037505 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4ed7b0-6a56-4d9f-a691-a111575e1c61" path="/var/lib/kubelet/pods/7d4ed7b0-6a56-4d9f-a691-a111575e1c61/volumes" Jul 12 17:04:45.198874 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:45.198841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:45.259072 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:45.259040 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:54.475454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:04:54.475423 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zjsj4" Jul 12 17:05:25.054745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.054709 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:25.155532 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.155497 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:25.359675 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.359598 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-756f7d7b75-hzwbf"] Jul 12 17:05:25.359870 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.359859 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4ed7b0-6a56-4d9f-a691-a111575e1c61" containerName="console" Jul 12 17:05:25.359935 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.359872 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4ed7b0-6a56-4d9f-a691-a111575e1c61" containerName="console" Jul 12 17:05:25.359974 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.359937 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4ed7b0-6a56-4d9f-a691-a111575e1c61" containerName="console" Jul 12 17:05:25.364105 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.364088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.373588 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.373565 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-756f7d7b75-hzwbf"] Jul 12 17:05:25.406816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.406784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:25.459416 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.459386 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:25.513218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-trusted-ca-bundle\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.513309 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-console-config\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.513309 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-serving-cert\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.513389 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-oauth-serving-cert\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.513389 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sbh\" (UniqueName: \"kubernetes.io/projected/d526e291-1569-4b11-a004-789f57269494-kube-api-access-k5sbh\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.513452 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-oauth-config\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.513452 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.513419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-service-ca\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614446 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-trusted-ca-bundle\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614446 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-console-config\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614446 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-serving-cert\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-oauth-serving-cert\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sbh\" (UniqueName: \"kubernetes.io/projected/d526e291-1569-4b11-a004-789f57269494-kube-api-access-k5sbh\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-oauth-config\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.614692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.614564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-service-ca\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.615266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.615241 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-console-config\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.615400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.615276 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-service-ca\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.615400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.615346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-oauth-serving-cert\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.615515 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.615461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-trusted-ca-bundle\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.616979 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.616952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-serving-cert\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.616979 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.616974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-oauth-config\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.622182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.622157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sbh\" (UniqueName: \"kubernetes.io/projected/d526e291-1569-4b11-a004-789f57269494-kube-api-access-k5sbh\") pod \"console-756f7d7b75-hzwbf\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.673780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.673747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:25.802454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:25.802423 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-756f7d7b75-hzwbf"] Jul 12 17:05:25.805245 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:05:25.805215 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd526e291_1569_4b11_a004_789f57269494.slice/crio-2a0419678d80946bd12588399a08afb9f1934682b10b7b4689db472e20cb33b1 WatchSource:0}: Error finding container 2a0419678d80946bd12588399a08afb9f1934682b10b7b4689db472e20cb33b1: Status 404 returned error can't find the container with id 2a0419678d80946bd12588399a08afb9f1934682b10b7b4689db472e20cb33b1 Jul 12 17:05:26.689394 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:26.689359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f7d7b75-hzwbf" event={"ID":"d526e291-1569-4b11-a004-789f57269494","Type":"ContainerStarted","Data":"614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df"} Jul 12 17:05:26.689763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:26.689406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f7d7b75-hzwbf" event={"ID":"d526e291-1569-4b11-a004-789f57269494","Type":"ContainerStarted","Data":"2a0419678d80946bd12588399a08afb9f1934682b10b7b4689db472e20cb33b1"} Jul 12 17:05:26.718829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:26.718771 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-756f7d7b75-hzwbf" podStartSLOduration=1.7187542260000002 podStartE2EDuration="1.718754226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:05:26.716581923 +0000 UTC m=+134.211488362" watchObservedRunningTime="2026-07-12 17:05:26.718754226 +0000 UTC m=+134.213660666" Jul 12 17:05:31.175777 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.175747 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 17:05:31.179526 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.179509 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.182288 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 12 17:05:31.182288 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 12 17:05:31.182288 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 12 17:05:31.182536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182298 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 12 17:05:31.182536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182307 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 12 17:05:31.182536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182266 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 12 17:05:31.182536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182300 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sbl8b\"" Jul 12 17:05:31.182536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182380 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 12 17:05:31.182840 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.182825 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 12 17:05:31.189117 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.189101 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 12 17:05:31.200312 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.200292 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 17:05:31.264548 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264673 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264673 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-web-config\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264673 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264673 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af34459a-a7f4-4e1e-9516-6450972d1240-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af34459a-a7f4-4e1e-9516-6450972d1240-config-out\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34459a-a7f4-4e1e-9516-6450972d1240-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cw64\" (UniqueName: \"kubernetes.io/projected/af34459a-a7f4-4e1e-9516-6450972d1240-kube-api-access-7cw64\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-config-volume\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af34459a-a7f4-4e1e-9516-6450972d1240-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.264999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.264893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af34459a-a7f4-4e1e-9516-6450972d1240-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.365901 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.365867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.365939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af34459a-a7f4-4e1e-9516-6450972d1240-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.365985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366028 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-web-config\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af34459a-a7f4-4e1e-9516-6450972d1240-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af34459a-a7f4-4e1e-9516-6450972d1240-config-out\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34459a-a7f4-4e1e-9516-6450972d1240-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cw64\" (UniqueName: \"kubernetes.io/projected/af34459a-a7f4-4e1e-9516-6450972d1240-kube-api-access-7cw64\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-config-volume\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.366525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af34459a-a7f4-4e1e-9516-6450972d1240-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.367073 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.366826 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af34459a-a7f4-4e1e-9516-6450972d1240-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.368204 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.367616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af34459a-a7f4-4e1e-9516-6450972d1240-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.368204 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.367895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af34459a-a7f4-4e1e-9516-6450972d1240-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.369114 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.369085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.369208 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.369165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.369272 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.369207 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.369272 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.369264 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.369379 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.369287 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-web-config\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.369624 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.369610 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af34459a-a7f4-4e1e-9516-6450972d1240-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.370031 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.370010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.370095 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.370029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af34459a-a7f4-4e1e-9516-6450972d1240-config-out\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.371151 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.371136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af34459a-a7f4-4e1e-9516-6450972d1240-config-volume\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.377035 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.377014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cw64\" (UniqueName: \"kubernetes.io/projected/af34459a-a7f4-4e1e-9516-6450972d1240-kube-api-access-7cw64\") pod \"alertmanager-main-0\" (UID: \"af34459a-a7f4-4e1e-9516-6450972d1240\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.489654 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.489554 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 12 17:05:31.622067 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.622042 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 17:05:31.624307 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:05:31.624277 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf34459a_a7f4_4e1e_9516_6450972d1240.slice/crio-72815a772e611bb91d68d8bf014e0c073e62ba1df0c483054f6a0d23254c6b59 WatchSource:0}: Error finding container 72815a772e611bb91d68d8bf014e0c073e62ba1df0c483054f6a0d23254c6b59: Status 404 returned error can't find the container with id 72815a772e611bb91d68d8bf014e0c073e62ba1df0c483054f6a0d23254c6b59 Jul 12 17:05:31.713428 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.713397 2573 generic.go:356] "Generic (PLEG): container finished" podID="af34459a-a7f4-4e1e-9516-6450972d1240" containerID="8fbf6075224ae79aed0e2bf04acea337f4a90dc33206c6c57a49e9b1448e8643" exitCode=0 Jul 12 17:05:31.713562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.713482 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerDied","Data":"8fbf6075224ae79aed0e2bf04acea337f4a90dc33206c6c57a49e9b1448e8643"} Jul 12 17:05:31.713627 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:31.713601 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"72815a772e611bb91d68d8bf014e0c073e62ba1df0c483054f6a0d23254c6b59"} Jul 12 17:05:33.724012 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.723977 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"06200ed0312be60aa040ee30a106180a5c8963c42db62bfad2ad1a6c5ebf5c91"} Jul 12 17:05:33.724369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.724047 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"467265c88ae5a3c268729509891da537c511aceee070c2e8534ddac03c109688"} Jul 12 17:05:33.724369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.724076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"58db5dec3782a3e6956b518f3519eefad7762ec5a01751af6511ba98c9dce96d"} Jul 12 17:05:33.724369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.724106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"6e40bcbc0fc6d9d6b1ba9700b5c274df2619e586109e779dce646bb3c9eba4b7"} Jul 12 17:05:33.724369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.724143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"c84232f7de8e5dc504ce5fa7956e810e9d053469daed1eb1ae2215c9d8082fe6"} Jul 12 17:05:33.724369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.724168 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af34459a-a7f4-4e1e-9516-6450972d1240","Type":"ContainerStarted","Data":"6f2882e90f366b0fe6fbcf8a311ad4a15fb67aea3966884c2a3d349c3eab82d9"} Jul 12 17:05:33.754946 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:33.754883 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.180253251 podStartE2EDuration="2.754867264s" totalImagesPullingTime="1.55942023s" totalInitContainerRuntime="15.193783ms" isStatefulPod=false podCreationTimestamp="2026-07-12 17:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:05:33.752699641 +0000 UTC m=+141.247606082" watchObservedRunningTime="2026-07-12 17:05:33.754867264 +0000 UTC m=+141.249773703" Jul 12 17:05:35.674593 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:35.674546 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:35.675014 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:35.674617 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:35.679720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:35.679698 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:35.735057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:35.735024 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:05:35.782332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:35.782290 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dcc7b7d9-jz2fw"] Jul 12 17:05:35.902999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:35.902964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:36.007424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:36.007330 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:36.064933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:36.064884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:36.970607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:36.970575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:37.070174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:37.070139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:39.652073 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:39.652040 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:46.036260 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:46.036225 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:46.134400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:46.134366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:47.122800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:47.122762 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:05:47.218381 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:05:47.218349 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:00.802464 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:00.802420 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54dcc7b7d9-jz2fw" podUID="89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" containerName="console" containerID="cri-o://d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d" gracePeriod=15 Jul 12 17:06:01.038501 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.038477 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:06:01.120192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-oauth-config\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120143 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-trusted-ca-bundle\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120164 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-oauth-serving-cert\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120198 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-service-ca\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120226 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbmcn\" (UniqueName: \"kubernetes.io/projected/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-kube-api-access-cbmcn\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120255 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-config\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120289 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-serving-cert\") pod \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\" (UID: \"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2\") " Jul 12 17:06:01.120651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120604 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:06:01.120651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120614 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:06:01.120651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120625 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-service-ca" (OuterVolumeSpecName: "service-ca") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:06:01.120651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.120636 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-config" (OuterVolumeSpecName: "console-config") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:06:01.122313 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.122292 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:06:01.122520 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.122504 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-kube-api-access-cbmcn" (OuterVolumeSpecName: "kube-api-access-cbmcn") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "kube-api-access-cbmcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:06:01.122560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.122544 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" (UID: "89f7c7a8-d7ec-4530-89cc-9efdf9d226b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:06:01.221599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221567 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-oauth-config\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.221599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221593 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-trusted-ca-bundle\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.221599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221602 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-oauth-serving-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.221805 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221612 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.221805 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221620 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmcn\" (UniqueName: \"kubernetes.io/projected/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-kube-api-access-cbmcn\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.221805 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221629 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-config\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.221805 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.221637 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2-console-serving-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:06:01.828768 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.828730 2573 generic.go:356] "Generic (PLEG): container finished" podID="89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" containerID="d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d" exitCode=0 Jul 12 17:06:01.829182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.828780 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dcc7b7d9-jz2fw" Jul 12 17:06:01.829182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.828813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dcc7b7d9-jz2fw" event={"ID":"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2","Type":"ContainerDied","Data":"d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d"} Jul 12 17:06:01.829182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.828864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dcc7b7d9-jz2fw" event={"ID":"89f7c7a8-d7ec-4530-89cc-9efdf9d226b2","Type":"ContainerDied","Data":"b6a0c1e3a5bf3a38fb6fe7aea2d38b26d274903feb90a41760691073cb7c6a20"} Jul 12 17:06:01.829182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.828885 2573 scope.go:117] "RemoveContainer" containerID="d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d" Jul 12 17:06:01.837072 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.837057 2573 scope.go:117] "RemoveContainer" containerID="d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d" Jul 12 17:06:01.837311 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:06:01.837294 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d\": container with ID starting with d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d not found: ID does not exist" containerID="d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d" Jul 12 17:06:01.837372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.837317 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d"} err="failed to get container status \"d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d\": rpc error: code = NotFound desc = could not find container \"d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d\": container with ID starting with d76866d5070649ad0a929a7431cbccc3f6d761b18bb8e3f02a3187187dedb35d not found: ID does not exist" Jul 12 17:06:01.848716 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.848695 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dcc7b7d9-jz2fw"] Jul 12 17:06:01.852218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:01.852197 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54dcc7b7d9-jz2fw"] Jul 12 17:06:03.037688 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:03.037651 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" path="/var/lib/kubelet/pods/89f7c7a8-d7ec-4530-89cc-9efdf9d226b2/volumes" Jul 12 17:06:18.789622 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:18.789586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:18.843207 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:18.843163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:28.825971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.825935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:28.849200 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.849163 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5"] Jul 12 17:06:28.849537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.849520 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" containerName="console" Jul 12 17:06:28.849608 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.849540 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" containerName="console" Jul 12 17:06:28.849659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.849637 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f7c7a8-d7ec-4530-89cc-9efdf9d226b2" containerName="console" Jul 12 17:06:28.853775 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.853754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.857683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.857655 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dbfcd\"" Jul 12 17:06:28.857683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.857669 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 12 17:06:28.857683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.857683 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 12 17:06:28.857956 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.857703 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 12 17:06:28.857956 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.857658 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 12 17:06:28.859251 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.859222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6zz\" (UniqueName: \"kubernetes.io/projected/f9c3a184-8bbd-41ce-9d06-a25b3e5edf30-kube-api-access-8x6zz\") pod \"managed-serviceaccount-addon-agent-85d694bc64-5mhh5\" (UID: \"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.859347 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.859287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9c3a184-8bbd-41ce-9d06-a25b3e5edf30-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85d694bc64-5mhh5\" (UID: \"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.862952 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.862930 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5"] Jul 12 17:06:28.923249 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.923173 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp"] Jul 12 17:06:28.924291 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.924263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:28.926489 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.926473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:28.929203 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.929184 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 12 17:06:28.937383 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.937360 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp"] Jul 12 17:06:28.960569 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.960530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6zz\" (UniqueName: \"kubernetes.io/projected/f9c3a184-8bbd-41ce-9d06-a25b3e5edf30-kube-api-access-8x6zz\") pod \"managed-serviceaccount-addon-agent-85d694bc64-5mhh5\" (UID: \"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.960569 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.960570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e5186c7-f743-496a-a90e-a963c181094e-klusterlet-config\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:28.960820 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.960624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5186c7-f743-496a-a90e-a963c181094e-tmp\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:28.960820 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.960659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9c3a184-8bbd-41ce-9d06-a25b3e5edf30-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85d694bc64-5mhh5\" (UID: \"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.960820 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.960777 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9km\" (UniqueName: \"kubernetes.io/projected/0e5186c7-f743-496a-a90e-a963c181094e-kube-api-access-jk9km\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:28.963247 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.963223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9c3a184-8bbd-41ce-9d06-a25b3e5edf30-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85d694bc64-5mhh5\" (UID: \"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.971330 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.971298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6zz\" (UniqueName: \"kubernetes.io/projected/f9c3a184-8bbd-41ce-9d06-a25b3e5edf30-kube-api-access-8x6zz\") pod \"managed-serviceaccount-addon-agent-85d694bc64-5mhh5\" (UID: \"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:28.990308 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:28.990274 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:29.061175 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.061136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9km\" (UniqueName: \"kubernetes.io/projected/0e5186c7-f743-496a-a90e-a963c181094e-kube-api-access-jk9km\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.061347 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.061192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e5186c7-f743-496a-a90e-a963c181094e-klusterlet-config\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.061347 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.061241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5186c7-f743-496a-a90e-a963c181094e-tmp\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.061653 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.061631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5186c7-f743-496a-a90e-a963c181094e-tmp\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.063772 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.063752 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e5186c7-f743-496a-a90e-a963c181094e-klusterlet-config\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.069221 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.069193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9km\" (UniqueName: \"kubernetes.io/projected/0e5186c7-f743-496a-a90e-a963c181094e-kube-api-access-jk9km\") pod \"klusterlet-addon-workmgr-75b7479dcc-mzmtp\" (UID: \"0e5186c7-f743-496a-a90e-a963c181094e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.178951 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.178896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" Jul 12 17:06:29.236939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.236878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:29.310407 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.310382 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5"] Jul 12 17:06:29.312552 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:06:29.312525 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9c3a184_8bbd_41ce_9d06_a25b3e5edf30.slice/crio-3fcf275edbfee0cdbfdc6f0c0cc4b30707c8f00762639e34bda0519421cb84ad WatchSource:0}: Error finding container 3fcf275edbfee0cdbfdc6f0c0cc4b30707c8f00762639e34bda0519421cb84ad: Status 404 returned error can't find the container with id 3fcf275edbfee0cdbfdc6f0c0cc4b30707c8f00762639e34bda0519421cb84ad Jul 12 17:06:29.366622 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.366499 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp"] Jul 12 17:06:29.369079 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:06:29.369055 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5186c7_f743_496a_a90e_a963c181094e.slice/crio-2fe7556a174b49175d17e73da6139ce1a8f7157c86dd649f144f5e8065474886 WatchSource:0}: Error finding container 2fe7556a174b49175d17e73da6139ce1a8f7157c86dd649f144f5e8065474886: Status 404 returned error can't find the container with id 2fe7556a174b49175d17e73da6139ce1a8f7157c86dd649f144f5e8065474886 Jul 12 17:06:29.809453 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.809409 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:29.866857 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.866821 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:29.936137 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.936093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" event={"ID":"0e5186c7-f743-496a-a90e-a963c181094e","Type":"ContainerStarted","Data":"2fe7556a174b49175d17e73da6139ce1a8f7157c86dd649f144f5e8065474886"} Jul 12 17:06:29.937282 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:29.937247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" event={"ID":"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30","Type":"ContainerStarted","Data":"3fcf275edbfee0cdbfdc6f0c0cc4b30707c8f00762639e34bda0519421cb84ad"} Jul 12 17:06:33.956160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:33.956119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" event={"ID":"0e5186c7-f743-496a-a90e-a963c181094e","Type":"ContainerStarted","Data":"0021bda77fca6c9a34f907f33e89dc5c851ec488e54af0b83f23ae0196acdf84"} Jul 12 17:06:33.956619 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:33.956317 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:33.957672 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:33.957646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" event={"ID":"f9c3a184-8bbd-41ce-9d06-a25b3e5edf30","Type":"ContainerStarted","Data":"f5a8d0f81b0448b1546c727cb1c895abdcb1dc582d587b14e6bdc239d5270c83"} Jul 12 17:06:33.958391 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:33.958376 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" Jul 12 17:06:33.971340 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:33.970789 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75b7479dcc-mzmtp" podStartSLOduration=2.064007844 podStartE2EDuration="5.97077117s" totalImagesPullingTime="3.906763326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:06:33.970372932 +0000 UTC m=+201.465279391" watchObservedRunningTime="2026-07-12 17:06:33.97077117 +0000 UTC m=+201.465677627" Jul 12 17:06:33.984090 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:33.984045 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85d694bc64-5mhh5" podStartSLOduration=2.086341557 podStartE2EDuration="5.984030113s" totalImagesPullingTime="3.897688556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:06:33.983367736 +0000 UTC m=+201.478274190" watchObservedRunningTime="2026-07-12 17:06:33.984030113 +0000 UTC m=+201.478936598" Jul 12 17:06:56.585147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:56.585102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:57.378298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:57.378240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:06:58.590355 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.590321 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4"] Jul 12 17:06:58.594938 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.594906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.597432 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.597397 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 12 17:06:58.597581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.597487 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 12 17:06:58.597581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.597536 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bkx2g\"" Jul 12 17:06:58.617296 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.617262 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4"] Jul 12 17:06:58.719930 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.719875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52l2\" (UniqueName: \"kubernetes.io/projected/86e54afa-3ac0-4782-b6fe-d68aa92f87e3-kube-api-access-j52l2\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8jd4\" (UID: \"86e54afa-3ac0-4782-b6fe-d68aa92f87e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.720111 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.719939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e54afa-3ac0-4782-b6fe-d68aa92f87e3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8jd4\" (UID: \"86e54afa-3ac0-4782-b6fe-d68aa92f87e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.820836 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.820796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52l2\" (UniqueName: \"kubernetes.io/projected/86e54afa-3ac0-4782-b6fe-d68aa92f87e3-kube-api-access-j52l2\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8jd4\" (UID: \"86e54afa-3ac0-4782-b6fe-d68aa92f87e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.821034 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.820852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e54afa-3ac0-4782-b6fe-d68aa92f87e3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8jd4\" (UID: \"86e54afa-3ac0-4782-b6fe-d68aa92f87e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.821308 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.821290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e54afa-3ac0-4782-b6fe-d68aa92f87e3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8jd4\" (UID: \"86e54afa-3ac0-4782-b6fe-d68aa92f87e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.829624 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.829598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52l2\" (UniqueName: \"kubernetes.io/projected/86e54afa-3ac0-4782-b6fe-d68aa92f87e3-kube-api-access-j52l2\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8jd4\" (UID: \"86e54afa-3ac0-4782-b6fe-d68aa92f87e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:58.904213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:58.904185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" Jul 12 17:06:59.038172 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:06:59.038138 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e54afa_3ac0_4782_b6fe_d68aa92f87e3.slice/crio-65b9fa3d79452a30e0b38965c246c0ac7eafa19c706bbf0b2696bd7a225a645b WatchSource:0}: Error finding container 65b9fa3d79452a30e0b38965c246c0ac7eafa19c706bbf0b2696bd7a225a645b: Status 404 returned error can't find the container with id 65b9fa3d79452a30e0b38965c246c0ac7eafa19c706bbf0b2696bd7a225a645b Jul 12 17:06:59.038788 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:59.038770 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4"] Jul 12 17:06:59.055300 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:06:59.055274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" event={"ID":"86e54afa-3ac0-4782-b6fe-d68aa92f87e3","Type":"ContainerStarted","Data":"65b9fa3d79452a30e0b38965c246c0ac7eafa19c706bbf0b2696bd7a225a645b"} Jul 12 17:07:00.196760 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:00.196716 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:00.283778 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:00.283737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:02.072061 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:02.072029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" event={"ID":"86e54afa-3ac0-4782-b6fe-d68aa92f87e3","Type":"ContainerStarted","Data":"de43b9803156ebbadb215e2cd5922726139ae1b06396a136fe1263c7c714b3b2"} Jul 12 17:07:02.094412 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:02.094343 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8jd4" podStartSLOduration=1.442184992 podStartE2EDuration="4.094323627s" totalImagesPullingTime="2.652138635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:07:02.09209667 +0000 UTC m=+229.587003109" watchObservedRunningTime="2026-07-12 17:07:02.094323627 +0000 UTC m=+229.589230067" Jul 12 17:07:02.553222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:02.553187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:02.664153 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:02.664116 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:04.935497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:04.935453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:05.135479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:05.135447 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:07.172300 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:07.172267 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:07.274774 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:07.274738 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:09.906100 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.906064 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5v62g"] Jul 12 17:07:09.909292 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.909273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:09.911668 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.911651 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 12 17:07:09.911814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.911795 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8cn52\"" Jul 12 17:07:09.912751 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.912737 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 12 17:07:09.917976 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.917955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5v62g"] Jul 12 17:07:09.923272 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.923229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5m8z\" (UniqueName: \"kubernetes.io/projected/d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5-kube-api-access-n5m8z\") pod \"cert-manager-webhook-6486ffd997-5v62g\" (UID: \"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:09.923362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:09.923332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5v62g\" (UID: \"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:10.024390 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.024361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5v62g\" (UID: \"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:10.024565 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.024421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5m8z\" (UniqueName: \"kubernetes.io/projected/d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5-kube-api-access-n5m8z\") pod \"cert-manager-webhook-6486ffd997-5v62g\" (UID: \"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:10.033439 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.033411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5v62g\" (UID: \"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:10.033547 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.033472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5m8z\" (UniqueName: \"kubernetes.io/projected/d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5-kube-api-access-n5m8z\") pod \"cert-manager-webhook-6486ffd997-5v62g\" (UID: \"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:10.219065 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.218980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:10.345895 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.345856 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5v62g"] Jul 12 17:07:10.348687 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:10.348659 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8a4d8ac_dbfd_4107_84bd_250b4d70cdf5.slice/crio-78847d1faef496657221bce1f67bb51ab4962b6ee69111f03342f2a55acb7dee WatchSource:0}: Error finding container 78847d1faef496657221bce1f67bb51ab4962b6ee69111f03342f2a55acb7dee: Status 404 returned error can't find the container with id 78847d1faef496657221bce1f67bb51ab4962b6ee69111f03342f2a55acb7dee Jul 12 17:07:10.490717 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:10.490632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:11.111345 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:11.111304 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" event={"ID":"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5","Type":"ContainerStarted","Data":"78847d1faef496657221bce1f67bb51ab4962b6ee69111f03342f2a55acb7dee"} Jul 12 17:07:12.147435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:12.147399 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:13.121819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.121784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" event={"ID":"d8a4d8ac-dbfd-4107-84bd-250b4d70cdf5","Type":"ContainerStarted","Data":"4f1527dcdd2d027fcf936e30ab78b89a0124ad074a9b9fbf33dd4b43ac7c18eb"} Jul 12 17:07:13.121993 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.121928 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:13.138724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.138675 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" podStartSLOduration=1.482695022 podStartE2EDuration="4.138658437s" totalImagesPullingTime="2.655963415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:07:13.136592595 +0000 UTC m=+240.631499034" watchObservedRunningTime="2026-07-12 17:07:13.138658437 +0000 UTC m=+240.633564876" Jul 12 17:07:13.252472 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.252430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:13.313182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.313108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:13.579158 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.579049 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:13.641205 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:13.641175 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:15.598354 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.598319 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb"] Jul 12 17:07:15.601616 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.601599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.604023 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.604003 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 12 17:07:15.604135 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.604024 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 12 17:07:15.605101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.605082 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5bpkb\"" Jul 12 17:07:15.609978 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.609958 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb"] Jul 12 17:07:15.672401 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.672363 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29v46\" (UniqueName: \"kubernetes.io/projected/89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad-kube-api-access-29v46\") pod \"openshift-lws-operator-fd8ccff4c-74phb\" (UID: \"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.672608 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.672511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-74phb\" (UID: \"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.773318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.773279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-74phb\" (UID: \"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.773490 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.773331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29v46\" (UniqueName: \"kubernetes.io/projected/89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad-kube-api-access-29v46\") pod \"openshift-lws-operator-fd8ccff4c-74phb\" (UID: \"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.773668 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.773650 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-74phb\" (UID: \"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.781322 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.781293 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29v46\" (UniqueName: \"kubernetes.io/projected/89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad-kube-api-access-29v46\") pod \"openshift-lws-operator-fd8ccff4c-74phb\" (UID: \"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:15.910599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:15.910566 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" Jul 12 17:07:16.037075 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:16.037041 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb"] Jul 12 17:07:16.040642 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:16.040611 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89eb31b7_b4f9_4f54_9caa_2e1b27d4cdad.slice/crio-0344c6c0744ed6ba56aeb2520e314d6d11ecf13fefb5b86e1abfa7f0335b9ae6 WatchSource:0}: Error finding container 0344c6c0744ed6ba56aeb2520e314d6d11ecf13fefb5b86e1abfa7f0335b9ae6: Status 404 returned error can't find the container with id 0344c6c0744ed6ba56aeb2520e314d6d11ecf13fefb5b86e1abfa7f0335b9ae6 Jul 12 17:07:16.135591 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:16.135559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" event={"ID":"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad","Type":"ContainerStarted","Data":"0344c6c0744ed6ba56aeb2520e314d6d11ecf13fefb5b86e1abfa7f0335b9ae6"} Jul 12 17:07:19.128628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:19.128599 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5v62g" Jul 12 17:07:19.150061 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:19.150028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" event={"ID":"89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad","Type":"ContainerStarted","Data":"2483555e61cad9946b8c1865326e3de57d6d2a2717b760600c8e1a6137900eea"} Jul 12 17:07:19.173723 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:19.173668 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-74phb" podStartSLOduration=1.7390919550000001 podStartE2EDuration="4.173646322s" totalImagesPullingTime="2.434554367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:07:19.17182244 +0000 UTC m=+246.666728893" watchObservedRunningTime="2026-07-12 17:07:19.173646322 +0000 UTC m=+246.668552762" Jul 12 17:07:19.911544 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:19.911512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:19.997771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:19.997743 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:22.001560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:22.001528 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:34.259368 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:34.259324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:34.344215 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:34.344170 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:37.585782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:37.585748 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:37.681526 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:37.681493 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:38.200607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.200575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:38.494172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.494092 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:38.611848 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.611812 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6"] Jul 12 17:07:38.613810 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.613776 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:38.620153 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.620132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.624254 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.624232 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 17:07:38.624386 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.624292 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 17:07:38.624468 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.624397 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6fnfz\"" Jul 12 17:07:38.624524 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.624515 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 17:07:38.624692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.624676 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 17:07:38.633276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.633251 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6"] Jul 12 17:07:38.757907 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.757819 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:38.779985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.779959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/408afd12-f329-4b1e-9228-4d40fc53d53a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.780071 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.779997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/408afd12-f329-4b1e-9228-4d40fc53d53a-manifests\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.780127 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.780076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408afd12-f329-4b1e-9228-4d40fc53d53a-tmp\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.780196 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.780175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/408afd12-f329-4b1e-9228-4d40fc53d53a-webhook-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.780235 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.780205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8xx\" (UniqueName: \"kubernetes.io/projected/408afd12-f329-4b1e-9228-4d40fc53d53a-kube-api-access-nj8xx\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.873598 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.873554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:38.881008 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.880980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/408afd12-f329-4b1e-9228-4d40fc53d53a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.881082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.881030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/408afd12-f329-4b1e-9228-4d40fc53d53a-manifests\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.881165 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.881095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408afd12-f329-4b1e-9228-4d40fc53d53a-tmp\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.881165 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.881141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/408afd12-f329-4b1e-9228-4d40fc53d53a-webhook-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.881233 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.881164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8xx\" (UniqueName: \"kubernetes.io/projected/408afd12-f329-4b1e-9228-4d40fc53d53a-kube-api-access-nj8xx\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.881549 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.881526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/408afd12-f329-4b1e-9228-4d40fc53d53a-manifests\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.883437 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.883412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408afd12-f329-4b1e-9228-4d40fc53d53a-tmp\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.883541 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.883508 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/408afd12-f329-4b1e-9228-4d40fc53d53a-webhook-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.883598 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.883582 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/408afd12-f329-4b1e-9228-4d40fc53d53a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.889780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.889748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8xx\" (UniqueName: \"kubernetes.io/projected/408afd12-f329-4b1e-9228-4d40fc53d53a-kube-api-access-nj8xx\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-hnhv6\" (UID: \"408afd12-f329-4b1e-9228-4d40fc53d53a\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.929809 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.929765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:38.966661 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:38.966625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:39.065087 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:39.065066 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6"] Jul 12 17:07:39.065935 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:39.065892 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:39.068201 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:39.068171 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408afd12_f329_4b1e_9228_4d40fc53d53a.slice/crio-80fef1dd782d9208fbf116c5d615ef73d1fc897c684174d4fa4ed7a62eccd02c WatchSource:0}: Error finding container 80fef1dd782d9208fbf116c5d615ef73d1fc897c684174d4fa4ed7a62eccd02c: Status 404 returned error can't find the container with id 80fef1dd782d9208fbf116c5d615ef73d1fc897c684174d4fa4ed7a62eccd02c Jul 12 17:07:39.164818 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:39.164784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:39.233043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:39.233003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" event={"ID":"408afd12-f329-4b1e-9228-4d40fc53d53a","Type":"ContainerStarted","Data":"80fef1dd782d9208fbf116c5d615ef73d1fc897c684174d4fa4ed7a62eccd02c"} Jul 12 17:07:42.249382 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:42.249292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" event={"ID":"408afd12-f329-4b1e-9228-4d40fc53d53a","Type":"ContainerStarted","Data":"3f0aa8d081836714d0ae51e014e70588a5322741646be3c0bd1309eafdf9d50d"} Jul 12 17:07:43.254760 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:43.254723 2573 generic.go:356] "Generic (PLEG): container finished" podID="408afd12-f329-4b1e-9228-4d40fc53d53a" containerID="3f0aa8d081836714d0ae51e014e70588a5322741646be3c0bd1309eafdf9d50d" exitCode=0 Jul 12 17:07:43.255270 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:43.254803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" event={"ID":"408afd12-f329-4b1e-9228-4d40fc53d53a","Type":"ContainerDied","Data":"3f0aa8d081836714d0ae51e014e70588a5322741646be3c0bd1309eafdf9d50d"} Jul 12 17:07:44.261410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:44.261364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" event={"ID":"408afd12-f329-4b1e-9228-4d40fc53d53a","Type":"ContainerStarted","Data":"deccf3959ece0dd6e1c348061fa1bd8fb6ecd1d92c68629d0378839ff84eb988"} Jul 12 17:07:44.261879 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:44.261529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:44.284152 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:44.284082 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" podStartSLOduration=2.039968252 podStartE2EDuration="6.284066954s" totalImagesPullingTime="3.25480358s" totalInitContainerRuntime="989.295122ms" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:07:44.283097164 +0000 UTC m=+271.778003604" watchObservedRunningTime="2026-07-12 17:07:44.284066954 +0000 UTC m=+271.778973393" Jul 12 17:07:50.332791 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:50.332754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:51.489900 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:51.489865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:55.267612 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:55.267583 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-hnhv6" Jul 12 17:07:57.362768 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:57.362730 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:57.491411 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:57.491370 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:57.777537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:57.777500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:58.053500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.053407 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59584446bf-wlcj8"] Jul 12 17:07:58.062155 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.062125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.067418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.067392 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 12 17:07:58.067725 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.067693 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dfm9d\"" Jul 12 17:07:58.067949 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.067904 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 12 17:07:58.068171 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.068147 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 17:07:58.068395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.068375 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 17:07:58.069952 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.069933 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59584446bf-wlcj8"] Jul 12 17:07:58.080560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.080531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:58.156159 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.156117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659b6479-271a-45c8-9a9f-8c1303778839-tls-certs\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.156361 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.156178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/659b6479-271a-45c8-9a9f-8c1303778839-tmp\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.156361 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.156293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd76s\" (UniqueName: \"kubernetes.io/projected/659b6479-271a-45c8-9a9f-8c1303778839-kube-api-access-gd76s\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.215134 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.215094 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b6nbq"] Jul 12 17:07:58.218462 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.218446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.222902 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.222868 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 12 17:07:58.223073 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.222986 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zbqns\"" Jul 12 17:07:58.240684 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.240656 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b6nbq"] Jul 12 17:07:58.257186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.256739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd76s\" (UniqueName: \"kubernetes.io/projected/659b6479-271a-45c8-9a9f-8c1303778839-kube-api-access-gd76s\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.257186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.256807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659b6479-271a-45c8-9a9f-8c1303778839-tls-certs\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.257186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.256842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/659b6479-271a-45c8-9a9f-8c1303778839-tmp\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.257186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.256875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f94e4be4-b2df-48d2-824c-4a136d96aca4-cert\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.257186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.256949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79q2\" (UniqueName: \"kubernetes.io/projected/f94e4be4-b2df-48d2-824c-4a136d96aca4-kube-api-access-n79q2\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.259600 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.259568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/659b6479-271a-45c8-9a9f-8c1303778839-tmp\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.259734 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.259654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659b6479-271a-45c8-9a9f-8c1303778839-tls-certs\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.270414 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.270392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd76s\" (UniqueName: \"kubernetes.io/projected/659b6479-271a-45c8-9a9f-8c1303778839-kube-api-access-gd76s\") pod \"kube-auth-proxy-59584446bf-wlcj8\" (UID: \"659b6479-271a-45c8-9a9f-8c1303778839\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.276296 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.276277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:58.357552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.357461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f94e4be4-b2df-48d2-824c-4a136d96aca4-cert\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.357552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.357506 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79q2\" (UniqueName: \"kubernetes.io/projected/f94e4be4-b2df-48d2-824c-4a136d96aca4-kube-api-access-n79q2\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.357733 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:07:58.357626 2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 12 17:07:58.357733 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:07:58.357701 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94e4be4-b2df-48d2-824c-4a136d96aca4-cert podName:f94e4be4-b2df-48d2-824c-4a136d96aca4 nodeName:}" failed. No retries permitted until 2026-07-12 17:07:58.857681328 +0000 UTC m=+286.352587766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f94e4be4-b2df-48d2-824c-4a136d96aca4-cert") pod "odh-model-controller-7998c67dc4-b6nbq" (UID: "f94e4be4-b2df-48d2-824c-4a136d96aca4") : secret "odh-model-controller-webhook-cert" not found Jul 12 17:07:58.369625 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.369595 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:58.372259 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.372236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" Jul 12 17:07:58.372727 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.372708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79q2\" (UniqueName: \"kubernetes.io/projected/f94e4be4-b2df-48d2-824c-4a136d96aca4-kube-api-access-n79q2\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.468639 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.468605 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:58.508559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.508510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59584446bf-wlcj8"] Jul 12 17:07:58.510778 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:58.510742 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659b6479_271a_45c8_9a9f_8c1303778839.slice/crio-86c10924078e249a69b95942ad7c51b5bbb149d6589d250965831c74ed73c9f9 WatchSource:0}: Error finding container 86c10924078e249a69b95942ad7c51b5bbb149d6589d250965831c74ed73c9f9: Status 404 returned error can't find the container with id 86c10924078e249a69b95942ad7c51b5bbb149d6589d250965831c74ed73c9f9 Jul 12 17:07:58.571498 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.571462 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:58.862900 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.862866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f94e4be4-b2df-48d2-824c-4a136d96aca4-cert\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:58.865553 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:58.865531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f94e4be4-b2df-48d2-824c-4a136d96aca4-cert\") pod \"odh-model-controller-7998c67dc4-b6nbq\" (UID: \"f94e4be4-b2df-48d2-824c-4a136d96aca4\") " pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:59.035523 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.035483 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:59.128866 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.128798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:07:59.159041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.158130 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pj8pm"] Jul 12 17:07:59.164303 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.163608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.167332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.167111 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ncspb\"" Jul 12 17:07:59.167467 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.167335 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 12 17:07:59.179653 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.178671 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pj8pm"] Jul 12 17:07:59.221177 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.221137 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:59.224866 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.224838 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-njdmc"] Jul 12 17:07:59.228398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.228378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.231243 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.231179 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 12 17:07:59.231243 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.231200 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 12 17:07:59.231401 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.231280 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 17:07:59.231401 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.231200 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h97ml\"" Jul 12 17:07:59.231573 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.231553 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 12 17:07:59.245445 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.245413 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-njdmc"] Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.266981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-token\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedfbf65-d32e-43b4-9758-d4635f2e3755-local-certs\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-cacerts\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtdv\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-kube-api-access-gdtdv\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267276 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb488a1-c450-4b2d-91eb-8f54f4c118c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-pj8pm\" (UID: \"ecb488a1-c450-4b2d-91eb-8f54f4c118c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.267480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.267327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvpl\" (UniqueName: \"kubernetes.io/projected/ecb488a1-c450-4b2d-91eb-8f54f4c118c2-kube-api-access-vtvpl\") pod \"model-serving-api-54d89bd5fc-pj8pm\" (UID: \"ecb488a1-c450-4b2d-91eb-8f54f4c118c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.305862 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.305839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:07:59.306963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.306944 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b6nbq"] Jul 12 17:07:59.308749 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:59.308726 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94e4be4_b2df_48d2_824c_4a136d96aca4.slice/crio-c447b3f4fea5129ec453bce353d5ef93e455a03db9bc08ebf49b2e9bd41c9e6c WatchSource:0}: Error finding container c447b3f4fea5129ec453bce353d5ef93e455a03db9bc08ebf49b2e9bd41c9e6c: Status 404 returned error can't find the container with id c447b3f4fea5129ec453bce353d5ef93e455a03db9bc08ebf49b2e9bd41c9e6c Jul 12 17:07:59.326841 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.326813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" event={"ID":"659b6479-271a-45c8-9a9f-8c1303778839","Type":"ContainerStarted","Data":"86c10924078e249a69b95942ad7c51b5bbb149d6589d250965831c74ed73c9f9"} Jul 12 17:07:59.328219 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.328192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" event={"ID":"f94e4be4-b2df-48d2-824c-4a136d96aca4","Type":"ContainerStarted","Data":"c447b3f4fea5129ec453bce353d5ef93e455a03db9bc08ebf49b2e9bd41c9e6c"} Jul 12 17:07:59.368603 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-cacerts\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.368782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtdv\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-kube-api-access-gdtdv\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.368782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.368782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb488a1-c450-4b2d-91eb-8f54f4c118c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-pj8pm\" (UID: \"ecb488a1-c450-4b2d-91eb-8f54f4c118c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.368782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvpl\" (UniqueName: \"kubernetes.io/projected/ecb488a1-c450-4b2d-91eb-8f54f4c118c2-kube-api-access-vtvpl\") pod \"model-serving-api-54d89bd5fc-pj8pm\" (UID: \"ecb488a1-c450-4b2d-91eb-8f54f4c118c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.368782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.368782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-token\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.369100 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedfbf65-d32e-43b4-9758-d4635f2e3755-local-certs\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.369100 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.368822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.369970 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.369626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.371666 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.371640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.372266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.372017 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-cacerts\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.372266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.372223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.372681 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.372439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb488a1-c450-4b2d-91eb-8f54f4c118c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-pj8pm\" (UID: \"ecb488a1-c450-4b2d-91eb-8f54f4c118c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.372773 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.372698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedfbf65-d32e-43b4-9758-d4635f2e3755-local-certs\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.379075 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.377815 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-token\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.379075 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.378435 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtdv\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-kube-api-access-gdtdv\") pod \"istiod-openshift-gateway-7574459656-njdmc\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.379075 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.378978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvpl\" (UniqueName: \"kubernetes.io/projected/ecb488a1-c450-4b2d-91eb-8f54f4c118c2-kube-api-access-vtvpl\") pod \"model-serving-api-54d89bd5fc-pj8pm\" (UID: \"ecb488a1-c450-4b2d-91eb-8f54f4c118c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.482063 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.482030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:07:59.540979 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.540948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:07:59.650312 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.650272 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pj8pm"] Jul 12 17:07:59.654114 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:59.654044 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb488a1_c450_4b2d_91eb_8f54f4c118c2.slice/crio-80f0cf7d8ef0fa190e48da4ed8c2f502338f1684299381a760a81c9dde35d476 WatchSource:0}: Error finding container 80f0cf7d8ef0fa190e48da4ed8c2f502338f1684299381a760a81c9dde35d476: Status 404 returned error can't find the container with id 80f0cf7d8ef0fa190e48da4ed8c2f502338f1684299381a760a81c9dde35d476 Jul 12 17:07:59.711750 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:07:59.711683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-njdmc"] Jul 12 17:07:59.713303 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:07:59.713275 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedfbf65_d32e_43b4_9758_d4635f2e3755.slice/crio-3dfd7f53021caeb1e966dfd2abe50a35e9dfb4a92f86c9790c4c0103da7de571 WatchSource:0}: Error finding container 3dfd7f53021caeb1e966dfd2abe50a35e9dfb4a92f86c9790c4c0103da7de571: Status 404 returned error can't find the container with id 3dfd7f53021caeb1e966dfd2abe50a35e9dfb4a92f86c9790c4c0103da7de571 Jul 12 17:08:00.214533 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:00.214471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:00.347466 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:00.347362 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" event={"ID":"ecb488a1-c450-4b2d-91eb-8f54f4c118c2","Type":"ContainerStarted","Data":"80f0cf7d8ef0fa190e48da4ed8c2f502338f1684299381a760a81c9dde35d476"} Jul 12 17:08:00.354713 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:00.354676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" event={"ID":"dedfbf65-d32e-43b4-9758-d4635f2e3755","Type":"ContainerStarted","Data":"3dfd7f53021caeb1e966dfd2abe50a35e9dfb4a92f86c9790c4c0103da7de571"} Jul 12 17:08:04.531299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:04.531268 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:08:04.531585 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:04.531334 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:08:05.305499 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.305456 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:05.386094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.386057 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" event={"ID":"659b6479-271a-45c8-9a9f-8c1303778839","Type":"ContainerStarted","Data":"720570d2849c17458aa5f69e3306911270fb38194bfe396b5f8b133d612fb791"} Jul 12 17:08:05.388057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.388025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" event={"ID":"dedfbf65-d32e-43b4-9758-d4635f2e3755","Type":"ContainerStarted","Data":"d5119dcc345f31ef747dce78b2bb83e890c1fc7e5038184986bd6e56320b9e53"} Jul 12 17:08:05.388197 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.388083 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:08:05.389650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.389622 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" event={"ID":"ecb488a1-c450-4b2d-91eb-8f54f4c118c2","Type":"ContainerStarted","Data":"22437d756c0a8f8566a9e511ab4000dce9b3663b6676ddd8cbe179bb845ed572"} Jul 12 17:08:05.389776 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.389697 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:08:05.390056 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.389955 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-njdmc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 17:08:05.390056 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.390007 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" podUID="dedfbf65-d32e-43b4-9758-d4635f2e3755" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 17:08:05.391110 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.391089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" event={"ID":"f94e4be4-b2df-48d2-824c-4a136d96aca4","Type":"ContainerStarted","Data":"8136c0a5906cbdee338fe740e49009bcba8dbfafa87bb54a5410aae703b3bb4e"} Jul 12 17:08:05.391257 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.391244 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:08:05.416547 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.416491 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59584446bf-wlcj8" podStartSLOduration=1.403509663 podStartE2EDuration="7.416476557s" totalImagesPullingTime="6.012966894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:05.41390221 +0000 UTC m=+292.908808649" watchObservedRunningTime="2026-07-12 17:08:05.416476557 +0000 UTC m=+292.911382995" Jul 12 17:08:05.439249 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.439199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:05.462100 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.462052 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" podStartSLOduration=1.646540083 podStartE2EDuration="6.462036093s" totalImagesPullingTime="4.81549601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:05.45916307 +0000 UTC m=+292.954069509" watchObservedRunningTime="2026-07-12 17:08:05.462036093 +0000 UTC m=+292.956942526" Jul 12 17:08:05.490163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.490106 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" podStartSLOduration=2.269312964 podStartE2EDuration="7.490090502s" totalImagesPullingTime="5.220777538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:05.487307494 +0000 UTC m=+292.982213934" watchObservedRunningTime="2026-07-12 17:08:05.490090502 +0000 UTC m=+292.984996975" Jul 12 17:08:05.506663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:05.506606 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" podStartSLOduration=1.638446096 podStartE2EDuration="6.50659242s" totalImagesPullingTime="4.868146324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:05.505811708 +0000 UTC m=+293.000718167" watchObservedRunningTime="2026-07-12 17:08:05.50659242 +0000 UTC m=+293.001498888" Jul 12 17:08:06.088346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:06.088310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:06.294176 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:06.294142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:06.395952 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:06.395851 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:08:06.536683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:06.536646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:09.430739 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.430701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:09.487076 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.487047 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh"] Jul 12 17:08:09.490950 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.490888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.494225 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.494201 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 12 17:08:09.494546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.494532 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7r256\"" Jul 12 17:08:09.503749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.503728 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh"] Jul 12 17:08:09.510648 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.510624 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h"] Jul 12 17:08:09.514609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.514588 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:09.517316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.517297 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hlk8n\"" Jul 12 17:08:09.517478 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.517463 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 12 17:08:09.525436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.525414 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h"] Jul 12 17:08:09.535767 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.535729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:09.569514 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.569481 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be76c4d0-6b33-45a8-b8f4-ec42379b8e70-cert\") pod \"kserve-controller-manager-d6f4b65b8-x2kwh\" (UID: \"be76c4d0-6b33-45a8-b8f4-ec42379b8e70\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.569666 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.569541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9zj\" (UniqueName: \"kubernetes.io/projected/be76c4d0-6b33-45a8-b8f4-ec42379b8e70-kube-api-access-nv9zj\") pod \"kserve-controller-manager-d6f4b65b8-x2kwh\" (UID: \"be76c4d0-6b33-45a8-b8f4-ec42379b8e70\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.569666 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.569568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prvs\" (UniqueName: \"kubernetes.io/projected/a0f6be29-8606-4180-aec2-5311c9416e90-kube-api-access-7prvs\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:09.569789 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.569717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0f6be29-8606-4180-aec2-5311c9416e90-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:09.666793 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.666756 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:09.671071 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.671048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9zj\" (UniqueName: \"kubernetes.io/projected/be76c4d0-6b33-45a8-b8f4-ec42379b8e70-kube-api-access-nv9zj\") pod \"kserve-controller-manager-d6f4b65b8-x2kwh\" (UID: \"be76c4d0-6b33-45a8-b8f4-ec42379b8e70\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.671206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.671083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prvs\" (UniqueName: \"kubernetes.io/projected/a0f6be29-8606-4180-aec2-5311c9416e90-kube-api-access-7prvs\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:09.671206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.671190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0f6be29-8606-4180-aec2-5311c9416e90-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:09.671371 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.671256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be76c4d0-6b33-45a8-b8f4-ec42379b8e70-cert\") pod \"kserve-controller-manager-d6f4b65b8-x2kwh\" (UID: \"be76c4d0-6b33-45a8-b8f4-ec42379b8e70\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.671371 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:08:09.671313 2573 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 12 17:08:09.671371 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:08:09.671372 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f6be29-8606-4180-aec2-5311c9416e90-cert podName:a0f6be29-8606-4180-aec2-5311c9416e90 nodeName:}" failed. No retries permitted until 2026-07-12 17:08:10.17135718 +0000 UTC m=+297.666263599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a0f6be29-8606-4180-aec2-5311c9416e90-cert") pod "llmisvc-controller-manager-777c6bd7d5-cll2h" (UID: "a0f6be29-8606-4180-aec2-5311c9416e90") : secret "llmisvc-webhook-server-cert" not found Jul 12 17:08:09.673606 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.673586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be76c4d0-6b33-45a8-b8f4-ec42379b8e70-cert\") pod \"kserve-controller-manager-d6f4b65b8-x2kwh\" (UID: \"be76c4d0-6b33-45a8-b8f4-ec42379b8e70\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.682263 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.682213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9zj\" (UniqueName: \"kubernetes.io/projected/be76c4d0-6b33-45a8-b8f4-ec42379b8e70-kube-api-access-nv9zj\") pod \"kserve-controller-manager-d6f4b65b8-x2kwh\" (UID: \"be76c4d0-6b33-45a8-b8f4-ec42379b8e70\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.683028 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.683008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prvs\" (UniqueName: \"kubernetes.io/projected/a0f6be29-8606-4180-aec2-5311c9416e90-kube-api-access-7prvs\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:09.804498 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.804460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:09.971346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:09.971317 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh"] Jul 12 17:08:09.973265 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:08:09.973238 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe76c4d0_6b33_45a8_b8f4_ec42379b8e70.slice/crio-f18cbde836ae70839a7fc04214d56a019d01eb07f6920532f93e0286d779c940 WatchSource:0}: Error finding container f18cbde836ae70839a7fc04214d56a019d01eb07f6920532f93e0286d779c940: Status 404 returned error can't find the container with id f18cbde836ae70839a7fc04214d56a019d01eb07f6920532f93e0286d779c940 Jul 12 17:08:10.036603 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.036568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:10.176567 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.176531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0f6be29-8606-4180-aec2-5311c9416e90-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:10.178833 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.178814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0f6be29-8606-4180-aec2-5311c9416e90-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cll2h\" (UID: \"a0f6be29-8606-4180-aec2-5311c9416e90\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:10.415302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.415269 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" event={"ID":"be76c4d0-6b33-45a8-b8f4-ec42379b8e70","Type":"ContainerStarted","Data":"f18cbde836ae70839a7fc04214d56a019d01eb07f6920532f93e0286d779c940"} Jul 12 17:08:10.427394 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.427368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:10.598086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.597966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h"] Jul 12 17:08:10.600736 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:08:10.600705 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda0f6be29_8606_4180_aec2_5311c9416e90.slice/crio-31bbbf5ea481613e7652a8825c7b77b43426cd2dafabf8151cf5ead3ff085f99 WatchSource:0}: Error finding container 31bbbf5ea481613e7652a8825c7b77b43426cd2dafabf8151cf5ead3ff085f99: Status 404 returned error can't find the container with id 31bbbf5ea481613e7652a8825c7b77b43426cd2dafabf8151cf5ead3ff085f99 Jul 12 17:08:10.706717 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.706649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:10.808622 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.808596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:10.914167 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:10.914137 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:11.422731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:11.422669 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" event={"ID":"a0f6be29-8606-4180-aec2-5311c9416e90","Type":"ContainerStarted","Data":"31bbbf5ea481613e7652a8825c7b77b43426cd2dafabf8151cf5ead3ff085f99"} Jul 12 17:08:13.045091 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.044893 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-njdmc"] Jul 12 17:08:13.086784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.045133 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" podUID="dedfbf65-d32e-43b4-9758-d4635f2e3755" containerName="discovery" containerID="cri-o://d5119dcc345f31ef747dce78b2bb83e890c1fc7e5038184986bd6e56320b9e53" gracePeriod=30 Jul 12 17:08:13.362626 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.362592 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:08:13.364160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.364136 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:08:13.433193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.433157 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" event={"ID":"be76c4d0-6b33-45a8-b8f4-ec42379b8e70","Type":"ContainerStarted","Data":"3b983fddad5de1e5c134e81170f09311ee1990ce7fe558b845e4e9364bb89711"} Jul 12 17:08:13.433318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.433233 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:13.434863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.434837 2573 generic.go:356] "Generic (PLEG): container finished" podID="dedfbf65-d32e-43b4-9758-d4635f2e3755" containerID="d5119dcc345f31ef747dce78b2bb83e890c1fc7e5038184986bd6e56320b9e53" exitCode=0 Jul 12 17:08:13.434999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.434875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" event={"ID":"dedfbf65-d32e-43b4-9758-d4635f2e3755","Type":"ContainerDied","Data":"d5119dcc345f31ef747dce78b2bb83e890c1fc7e5038184986bd6e56320b9e53"} Jul 12 17:08:13.450081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.450038 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" podStartSLOduration=1.956804225 podStartE2EDuration="4.450020076s" totalImagesPullingTime="2.493215851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:13.448666934 +0000 UTC m=+300.943573373" watchObservedRunningTime="2026-07-12 17:08:13.450020076 +0000 UTC m=+300.944926515" Jul 12 17:08:13.511068 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.511049 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:08:13.612962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.612864 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-kubeconfig\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.612962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.612907 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-dns-cert\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.613169 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.612982 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedfbf65-d32e-43b4-9758-d4635f2e3755-local-certs\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.613169 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.613025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-cacerts\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.613169 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.613050 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-ca-configmap\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.613169 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.613121 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdtdv\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-kube-api-access-gdtdv\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.613169 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.613156 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-token\") pod \"dedfbf65-d32e-43b4-9758-d4635f2e3755\" (UID: \"dedfbf65-d32e-43b4-9758-d4635f2e3755\") " Jul 12 17:08:13.613529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.613501 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:08:13.615400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.615369 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:08:13.615528 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.615395 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-cacerts" (OuterVolumeSpecName: "cacerts") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:08:13.615528 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.615475 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:08:13.615641 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.615534 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedfbf65-d32e-43b4-9758-d4635f2e3755-local-certs" (OuterVolumeSpecName: "local-certs") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:08:13.615641 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.615575 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-kube-api-access-gdtdv" (OuterVolumeSpecName: "kube-api-access-gdtdv") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "kube-api-access-gdtdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:08:13.615756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.615740 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-token" (OuterVolumeSpecName: "istio-token") pod "dedfbf65-d32e-43b4-9758-d4635f2e3755" (UID: "dedfbf65-d32e-43b4-9758-d4635f2e3755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:08:13.714088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714052 2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedfbf65-d32e-43b4-9758-d4635f2e3755-local-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:13.714088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714079 2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-cacerts\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:13.714088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714090 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-ca-configmap\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:13.714088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714098 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdtdv\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-kube-api-access-gdtdv\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:13.714334 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714108 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:13.714334 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714115 2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-kubeconfig\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:13.714334 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:13.714123 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dedfbf65-d32e-43b4-9758-d4635f2e3755-istio-csr-dns-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:08:14.441612 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.441574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" event={"ID":"a0f6be29-8606-4180-aec2-5311c9416e90","Type":"ContainerStarted","Data":"4fd71e9b3092f9c6d77a8942303f27a201fb6f95f289b222796533c4bc89f1e8"} Jul 12 17:08:14.442137 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.441693 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:14.443000 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.442977 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" Jul 12 17:08:14.443121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.443010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-njdmc" event={"ID":"dedfbf65-d32e-43b4-9758-d4635f2e3755","Type":"ContainerDied","Data":"3dfd7f53021caeb1e966dfd2abe50a35e9dfb4a92f86c9790c4c0103da7de571"} Jul 12 17:08:14.443121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.443055 2573 scope.go:117] "RemoveContainer" containerID="d5119dcc345f31ef747dce78b2bb83e890c1fc7e5038184986bd6e56320b9e53" Jul 12 17:08:14.479891 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.479836 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" podStartSLOduration=2.694510444 podStartE2EDuration="5.479820908s" totalImagesPullingTime="2.785310464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:14.479344923 +0000 UTC m=+301.974251364" watchObservedRunningTime="2026-07-12 17:08:14.479820908 +0000 UTC m=+301.974727348" Jul 12 17:08:14.501289 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.501261 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-njdmc"] Jul 12 17:08:14.502867 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:14.502847 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-njdmc"] Jul 12 17:08:15.039852 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:15.039816 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dedfbf65-d32e-43b4-9758-d4635f2e3755" path="/var/lib/kubelet/pods/dedfbf65-d32e-43b4-9758-d4635f2e3755/volumes" Jul 12 17:08:16.397575 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:16.397542 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-b6nbq" Jul 12 17:08:16.399529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:16.399512 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pj8pm" Jul 12 17:08:29.352296 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:29.352260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:29.407286 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:29.407255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:31.307211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:31.307156 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:31.405852 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:31.405817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:44.448441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:44.448409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x2kwh" Jul 12 17:08:45.451559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:45.451528 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cll2h" Jul 12 17:08:58.836015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:58.835979 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:58.933120 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:58.933085 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:58.988387 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:58.988352 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvfr9"] Jul 12 17:08:58.988813 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:58.988795 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedfbf65-d32e-43b4-9758-d4635f2e3755" containerName="discovery" Jul 12 17:08:58.988813 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:58.988813 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedfbf65-d32e-43b4-9758-d4635f2e3755" containerName="discovery" Jul 12 17:08:58.988989 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:58.988875 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedfbf65-d32e-43b4-9758-d4635f2e3755" containerName="discovery" Jul 12 17:08:59.001522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.001495 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:08:59.005072 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.005044 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xtct4\"" Jul 12 17:08:59.006357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.006338 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 12 17:08:59.006554 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.006527 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 12 17:08:59.008764 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.008739 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvfr9"] Jul 12 17:08:59.110717 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.110638 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:08:59.116623 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.116600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zz9\" (UniqueName: \"kubernetes.io/projected/e6e2d62c-28d9-4e0d-97b0-3599418d7d90-kube-api-access-r2zz9\") pod \"kuadrant-operator-catalog-lvfr9\" (UID: \"e6e2d62c-28d9-4e0d-97b0-3599418d7d90\") " pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:08:59.218046 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.218003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zz9\" (UniqueName: \"kubernetes.io/projected/e6e2d62c-28d9-4e0d-97b0-3599418d7d90-kube-api-access-r2zz9\") pod \"kuadrant-operator-catalog-lvfr9\" (UID: \"e6e2d62c-28d9-4e0d-97b0-3599418d7d90\") " pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:08:59.227532 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.227503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zz9\" (UniqueName: \"kubernetes.io/projected/e6e2d62c-28d9-4e0d-97b0-3599418d7d90-kube-api-access-r2zz9\") pod \"kuadrant-operator-catalog-lvfr9\" (UID: \"e6e2d62c-28d9-4e0d-97b0-3599418d7d90\") " pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:08:59.312369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.312329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:08:59.329416 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.329386 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvfr9"] Jul 12 17:08:59.448059 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.448035 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvfr9"] Jul 12 17:08:59.452602 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:08:59.452552 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e2d62c_28d9_4e0d_97b0_3599418d7d90.slice/crio-017f7ebf3b18bbe90c8eb263d66999a2a76ed38ebded997f546b2c4a10d9d1db WatchSource:0}: Error finding container 017f7ebf3b18bbe90c8eb263d66999a2a76ed38ebded997f546b2c4a10d9d1db: Status 404 returned error can't find the container with id 017f7ebf3b18bbe90c8eb263d66999a2a76ed38ebded997f546b2c4a10d9d1db Jul 12 17:08:59.453865 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.453443 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 17:08:59.539160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.539126 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qskq6"] Jul 12 17:08:59.543847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.543829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:08:59.550478 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.550453 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qskq6"] Jul 12 17:08:59.621429 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.621336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r2sm\" (UniqueName: \"kubernetes.io/projected/7b61b4a9-bc14-4bff-9266-ad5c5b962c38-kube-api-access-7r2sm\") pod \"kuadrant-operator-catalog-qskq6\" (UID: \"7b61b4a9-bc14-4bff-9266-ad5c5b962c38\") " pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:08:59.651238 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.651200 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" event={"ID":"e6e2d62c-28d9-4e0d-97b0-3599418d7d90","Type":"ContainerStarted","Data":"017f7ebf3b18bbe90c8eb263d66999a2a76ed38ebded997f546b2c4a10d9d1db"} Jul 12 17:08:59.722133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.722101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r2sm\" (UniqueName: \"kubernetes.io/projected/7b61b4a9-bc14-4bff-9266-ad5c5b962c38-kube-api-access-7r2sm\") pod \"kuadrant-operator-catalog-qskq6\" (UID: \"7b61b4a9-bc14-4bff-9266-ad5c5b962c38\") " pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:08:59.730003 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.729967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r2sm\" (UniqueName: \"kubernetes.io/projected/7b61b4a9-bc14-4bff-9266-ad5c5b962c38-kube-api-access-7r2sm\") pod \"kuadrant-operator-catalog-qskq6\" (UID: \"7b61b4a9-bc14-4bff-9266-ad5c5b962c38\") " pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:08:59.853853 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.853813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:08:59.983744 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:08:59.983723 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qskq6"] Jul 12 17:08:59.985434 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:08:59.985406 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b61b4a9_bc14_4bff_9266_ad5c5b962c38.slice/crio-defc9a7092db23b8534e1ac9a52d8cfbfbee35ed2a614b128f2531e02cdf1a44 WatchSource:0}: Error finding container defc9a7092db23b8534e1ac9a52d8cfbfbee35ed2a614b128f2531e02cdf1a44: Status 404 returned error can't find the container with id defc9a7092db23b8534e1ac9a52d8cfbfbee35ed2a614b128f2531e02cdf1a44 Jul 12 17:09:00.661011 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:00.660976 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" event={"ID":"7b61b4a9-bc14-4bff-9266-ad5c5b962c38","Type":"ContainerStarted","Data":"defc9a7092db23b8534e1ac9a52d8cfbfbee35ed2a614b128f2531e02cdf1a44"} Jul 12 17:09:01.673082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:01.673042 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" podUID="e6e2d62c-28d9-4e0d-97b0-3599418d7d90" containerName="registry-server" containerID="cri-o://65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49" gracePeriod=2 Jul 12 17:09:01.690075 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:01.689692 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" podStartSLOduration=1.551848955 podStartE2EDuration="3.689676866s" totalImagesPullingTime="2.137827911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:01.688114146 +0000 UTC m=+349.183020588" watchObservedRunningTime="2026-07-12 17:09:01.689676866 +0000 UTC m=+349.184583306" Jul 12 17:09:01.704881 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:01.704826 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" podStartSLOduration=1.096157551 podStartE2EDuration="2.704807176s" totalImagesPullingTime="1.608649625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:01.702862894 +0000 UTC m=+349.197769333" watchObservedRunningTime="2026-07-12 17:09:01.704807176 +0000 UTC m=+349.199713616" Jul 12 17:09:02.681356 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:02.681319 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" event={"ID":"7b61b4a9-bc14-4bff-9266-ad5c5b962c38","Type":"ContainerStarted","Data":"1e4903c36393818ebbfdd0261f9f3f5b6da1cb1a7af836a1627d2e958089c650"} Jul 12 17:09:02.682798 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:02.682778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" event={"ID":"e6e2d62c-28d9-4e0d-97b0-3599418d7d90","Type":"ContainerStarted","Data":"65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49"} Jul 12 17:09:04.312207 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.312182 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lvfr9_e6e2d62c-28d9-4e0d-97b0-3599418d7d90/registry-server/0.log" Jul 12 17:09:04.314554 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.314539 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:09:04.366771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.366729 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2zz9\" (UniqueName: \"kubernetes.io/projected/e6e2d62c-28d9-4e0d-97b0-3599418d7d90-kube-api-access-r2zz9\") pod \"e6e2d62c-28d9-4e0d-97b0-3599418d7d90\" (UID: \"e6e2d62c-28d9-4e0d-97b0-3599418d7d90\") " Jul 12 17:09:04.368905 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.368881 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e2d62c-28d9-4e0d-97b0-3599418d7d90-kube-api-access-r2zz9" (OuterVolumeSpecName: "kube-api-access-r2zz9") pod "e6e2d62c-28d9-4e0d-97b0-3599418d7d90" (UID: "e6e2d62c-28d9-4e0d-97b0-3599418d7d90"). InnerVolumeSpecName "kube-api-access-r2zz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:09:04.468321 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.468236 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zz9\" (UniqueName: \"kubernetes.io/projected/e6e2d62c-28d9-4e0d-97b0-3599418d7d90-kube-api-access-r2zz9\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:04.694245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.694172 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lvfr9_e6e2d62c-28d9-4e0d-97b0-3599418d7d90/registry-server/0.log" Jul 12 17:09:04.694245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.694214 2573 generic.go:356] "Generic (PLEG): container finished" podID="e6e2d62c-28d9-4e0d-97b0-3599418d7d90" containerID="65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49" exitCode=137 Jul 12 17:09:04.694482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.694261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" event={"ID":"e6e2d62c-28d9-4e0d-97b0-3599418d7d90","Type":"ContainerDied","Data":"65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49"} Jul 12 17:09:04.694482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.694278 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" Jul 12 17:09:04.694482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.694300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lvfr9" event={"ID":"e6e2d62c-28d9-4e0d-97b0-3599418d7d90","Type":"ContainerDied","Data":"017f7ebf3b18bbe90c8eb263d66999a2a76ed38ebded997f546b2c4a10d9d1db"} Jul 12 17:09:04.694482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.694319 2573 scope.go:117] "RemoveContainer" containerID="65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49" Jul 12 17:09:04.703783 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.703761 2573 scope.go:117] "RemoveContainer" containerID="65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49" Jul 12 17:09:04.704080 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:04.704061 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49\": container with ID starting with 65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49 not found: ID does not exist" containerID="65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49" Jul 12 17:09:04.704143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.704093 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49"} err="failed to get container status \"65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49\": rpc error: code = NotFound desc = could not find container \"65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49\": container with ID starting with 65f6750ac6d8c58aa6db09c5cf4d2b567b1383ff8dffc3290c213b50d77caa49 not found: ID does not exist" Jul 12 17:09:04.716506 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.716472 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvfr9"] Jul 12 17:09:04.723599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:04.723536 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvfr9"] Jul 12 17:09:05.038596 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:05.038519 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e2d62c-28d9-4e0d-97b0-3599418d7d90" path="/var/lib/kubelet/pods/e6e2d62c-28d9-4e0d-97b0-3599418d7d90/volumes" Jul 12 17:09:09.854679 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:09.854642 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:09:09.855128 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:09.854721 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:09:09.877691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:09.877660 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:09:10.750135 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:10.750107 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-qskq6" Jul 12 17:09:20.188712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.188680 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c66b7b8bf-gqvfm"] Jul 12 17:09:20.189181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.189059 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e2d62c-28d9-4e0d-97b0-3599418d7d90" containerName="registry-server" Jul 12 17:09:20.189181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.189070 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e2d62c-28d9-4e0d-97b0-3599418d7d90" containerName="registry-server" Jul 12 17:09:20.189181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.189143 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e2d62c-28d9-4e0d-97b0-3599418d7d90" containerName="registry-server" Jul 12 17:09:20.193576 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.193557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.205424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.205401 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c66b7b8bf-gqvfm"] Jul 12 17:09:20.301785 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.301755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfc4\" (UniqueName: \"kubernetes.io/projected/914ef91a-2af5-4803-88a9-fb54cbae72e7-kube-api-access-rxfc4\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.301971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.301794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-trusted-ca-bundle\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.301971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.301819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-oauth-serving-cert\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.301971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.301872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-serving-cert\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.301971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.301927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-oauth-config\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.302124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.302002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-config\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.302124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.302037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-service-ca\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.324363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.324335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:20.402510 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-oauth-serving-cert\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.402655 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-serving-cert\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.402655 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-oauth-config\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.402724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-config\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.402763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-service-ca\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.402819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfc4\" (UniqueName: \"kubernetes.io/projected/914ef91a-2af5-4803-88a9-fb54cbae72e7-kube-api-access-rxfc4\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.402872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.402850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-trusted-ca-bundle\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.403377 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.403346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-oauth-serving-cert\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.403564 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.403540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-config\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.403619 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.403541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-service-ca\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.403619 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.403597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914ef91a-2af5-4803-88a9-fb54cbae72e7-trusted-ca-bundle\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.405094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.405069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-oauth-config\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.405206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.405188 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914ef91a-2af5-4803-88a9-fb54cbae72e7-console-serving-cert\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.413334 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.413311 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfc4\" (UniqueName: \"kubernetes.io/projected/914ef91a-2af5-4803-88a9-fb54cbae72e7-kube-api-access-rxfc4\") pod \"console-5c66b7b8bf-gqvfm\" (UID: \"914ef91a-2af5-4803-88a9-fb54cbae72e7\") " pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.431792 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.431771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:20.503232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.503135 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:20.637065 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.637029 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c66b7b8bf-gqvfm"] Jul 12 17:09:20.642681 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:20.642626 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914ef91a_2af5_4803_88a9_fb54cbae72e7.slice/crio-1ec30ae79702114a1b833316b09cc83f0a2a49329e684a15656aa2529e2ea642 WatchSource:0}: Error finding container 1ec30ae79702114a1b833316b09cc83f0a2a49329e684a15656aa2529e2ea642: Status 404 returned error can't find the container with id 1ec30ae79702114a1b833316b09cc83f0a2a49329e684a15656aa2529e2ea642 Jul 12 17:09:20.776039 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.775966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c66b7b8bf-gqvfm" event={"ID":"914ef91a-2af5-4803-88a9-fb54cbae72e7","Type":"ContainerStarted","Data":"f10d59ca6f01d11c666124222701e6576c9f9228116541f683ba4de26f6170c2"} Jul 12 17:09:20.776039 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.776015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c66b7b8bf-gqvfm" event={"ID":"914ef91a-2af5-4803-88a9-fb54cbae72e7","Type":"ContainerStarted","Data":"1ec30ae79702114a1b833316b09cc83f0a2a49329e684a15656aa2529e2ea642"} Jul 12 17:09:20.796418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.796369 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c66b7b8bf-gqvfm" podStartSLOduration=0.796352861 podStartE2EDuration="796.352861ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:20.7954433 +0000 UTC m=+368.290349773" watchObservedRunningTime="2026-07-12 17:09:20.796352861 +0000 UTC m=+368.291259370" Jul 12 17:09:20.984533 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:20.984491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:21.041316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:21.041243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:22.665014 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:22.664977 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:25.320651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.320616 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t"] Jul 12 17:09:25.324332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.324312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:25.327257 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.327233 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lx7mr\"" Jul 12 17:09:25.332314 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.332287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t"] Jul 12 17:09:25.452162 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.452123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrns\" (UniqueName: \"kubernetes.io/projected/185f5e63-882b-422a-96e3-3d5c42abfcbf-kube-api-access-mzrns\") pod \"limitador-operator-controller-manager-859d7b78bd-zp58t\" (UID: \"185f5e63-882b-422a-96e3-3d5c42abfcbf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:25.552644 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.552610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrns\" (UniqueName: \"kubernetes.io/projected/185f5e63-882b-422a-96e3-3d5c42abfcbf-kube-api-access-mzrns\") pod \"limitador-operator-controller-manager-859d7b78bd-zp58t\" (UID: \"185f5e63-882b-422a-96e3-3d5c42abfcbf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:25.577963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.577890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrns\" (UniqueName: \"kubernetes.io/projected/185f5e63-882b-422a-96e3-3d5c42abfcbf-kube-api-access-mzrns\") pod \"limitador-operator-controller-manager-859d7b78bd-zp58t\" (UID: \"185f5e63-882b-422a-96e3-3d5c42abfcbf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:25.636756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.636723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:25.766430 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.766407 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t"] Jul 12 17:09:25.768764 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:25.768736 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185f5e63_882b_422a_96e3_3d5c42abfcbf.slice/crio-c1e308819be2791ef24ab9a5c352086abf5ae49e29078934e54d60a1de1544ed WatchSource:0}: Error finding container c1e308819be2791ef24ab9a5c352086abf5ae49e29078934e54d60a1de1544ed: Status 404 returned error can't find the container with id c1e308819be2791ef24ab9a5c352086abf5ae49e29078934e54d60a1de1544ed Jul 12 17:09:25.799749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:25.799716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" event={"ID":"185f5e63-882b-422a-96e3-3d5c42abfcbf","Type":"ContainerStarted","Data":"c1e308819be2791ef24ab9a5c352086abf5ae49e29078934e54d60a1de1544ed"} Jul 12 17:09:26.287213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:26.287185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:26.392008 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:26.391970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:27.812609 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:27.812577 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" event={"ID":"185f5e63-882b-422a-96e3-3d5c42abfcbf","Type":"ContainerStarted","Data":"eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4"} Jul 12 17:09:27.812984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:27.812655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:27.832360 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:27.832274 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" podStartSLOduration=1.035501281 podStartE2EDuration="2.832259645s" totalImagesPullingTime="1.796758364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:27.831428814 +0000 UTC m=+375.326335255" watchObservedRunningTime="2026-07-12 17:09:27.832259645 +0000 UTC m=+375.327166084" Jul 12 17:09:30.503353 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.503316 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:30.503724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.503407 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:30.508185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.508163 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:30.638524 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.638488 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc"] Jul 12 17:09:30.643147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.643106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.646186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.646164 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wxsvq\"" Jul 12 17:09:30.658432 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.658405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc"] Jul 12 17:09:30.798806 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.798718 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797fz\" (UniqueName: \"kubernetes.io/projected/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-kube-api-access-797fz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.798971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.798816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.829799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.829774 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c66b7b8bf-gqvfm" Jul 12 17:09:30.881834 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.881798 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-756f7d7b75-hzwbf"] Jul 12 17:09:30.899683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.899655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797fz\" (UniqueName: \"kubernetes.io/projected/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-kube-api-access-797fz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.899840 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.899749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.900247 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.900225 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.918354 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.918331 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-797fz\" (UniqueName: \"kubernetes.io/projected/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-kube-api-access-797fz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:30.953741 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:30.953712 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:31.024245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.024202 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:31.090184 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.090097 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc"] Jul 12 17:09:31.092467 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.092445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:31.093876 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:31.093848 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b3741c_5ea0_4b34_b992_57c3ae4dcbf3.slice/crio-fa0afbd9aa64388a5f01908873fb6ad3f619e61c66cba407809927eec220f3a7 WatchSource:0}: Error finding container fa0afbd9aa64388a5f01908873fb6ad3f619e61c66cba407809927eec220f3a7: Status 404 returned error can't find the container with id fa0afbd9aa64388a5f01908873fb6ad3f619e61c66cba407809927eec220f3a7 Jul 12 17:09:31.206795 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.206750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:31.676463 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.676427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:31.831456 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.831420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" event={"ID":"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3","Type":"ContainerStarted","Data":"fa0afbd9aa64388a5f01908873fb6ad3f619e61c66cba407809927eec220f3a7"} Jul 12 17:09:31.902375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:31.902344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:33.871399 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:33.871361 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs"] Jul 12 17:09:33.875000 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:33.874977 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:33.877662 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:33.877640 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 12 17:09:33.877799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:33.877637 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-swkpp\"" Jul 12 17:09:33.884709 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:33.884670 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs"] Jul 12 17:09:33.927017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:33.926974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4pbz\" (UniqueName: \"kubernetes.io/projected/45a23379-cdf6-4b51-b79f-a3945004016a-kube-api-access-w4pbz\") pod \"dns-operator-controller-manager-864dd78c4c-7nlfs\" (UID: \"45a23379-cdf6-4b51-b79f-a3945004016a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:34.028158 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.028119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4pbz\" (UniqueName: \"kubernetes.io/projected/45a23379-cdf6-4b51-b79f-a3945004016a-kube-api-access-w4pbz\") pod \"dns-operator-controller-manager-864dd78c4c-7nlfs\" (UID: \"45a23379-cdf6-4b51-b79f-a3945004016a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:34.036330 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.036301 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4pbz\" (UniqueName: \"kubernetes.io/projected/45a23379-cdf6-4b51-b79f-a3945004016a-kube-api-access-w4pbz\") pod \"dns-operator-controller-manager-864dd78c4c-7nlfs\" (UID: \"45a23379-cdf6-4b51-b79f-a3945004016a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:34.187584 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.187547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:34.285566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.285534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:34.322976 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.322941 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs"] Jul 12 17:09:34.325973 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:34.325944 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a23379_cdf6_4b51_b79f_a3945004016a.slice/crio-8dcb919df715d01bf6b4ff312c4ba4a84c4f1bd8b69d20b0886b4cc06bee97a3 WatchSource:0}: Error finding container 8dcb919df715d01bf6b4ff312c4ba4a84c4f1bd8b69d20b0886b4cc06bee97a3: Status 404 returned error can't find the container with id 8dcb919df715d01bf6b4ff312c4ba4a84c4f1bd8b69d20b0886b4cc06bee97a3 Jul 12 17:09:34.517731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.517655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:34.667710 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.667678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:34.850357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:34.850285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" event={"ID":"45a23379-cdf6-4b51-b79f-a3945004016a","Type":"ContainerStarted","Data":"8dcb919df715d01bf6b4ff312c4ba4a84c4f1bd8b69d20b0886b4cc06bee97a3"} Jul 12 17:09:36.867863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:36.867825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" event={"ID":"45a23379-cdf6-4b51-b79f-a3945004016a","Type":"ContainerStarted","Data":"f03ffb3eec4a98742535af0f304b59f8028c966971861f42f7459b607afed23c"} Jul 12 17:09:36.868330 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:36.867952 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:36.888037 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:36.887987 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" podStartSLOduration=2.139622933 podStartE2EDuration="3.887973567s" totalImagesPullingTime="1.748350634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:36.886575077 +0000 UTC m=+384.381481522" watchObservedRunningTime="2026-07-12 17:09:36.887973567 +0000 UTC m=+384.382880020" Jul 12 17:09:36.923850 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:36.923810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:37.027331 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.027297 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:37.326016 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.325979 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw"] Jul 12 17:09:37.330142 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.330114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:37.333259 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.333234 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4jmzk\"" Jul 12 17:09:37.340762 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.340732 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw"] Jul 12 17:09:37.365266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.365235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9z8\" (UniqueName: \"kubernetes.io/projected/fa51c4cf-ab54-439e-a656-a7f2cfa06e6b-kube-api-access-9b9z8\") pod \"authorino-operator-75cc8d5cb7-5hcqw\" (UID: \"fa51c4cf-ab54-439e-a656-a7f2cfa06e6b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:37.466402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.466364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9z8\" (UniqueName: \"kubernetes.io/projected/fa51c4cf-ab54-439e-a656-a7f2cfa06e6b-kube-api-access-9b9z8\") pod \"authorino-operator-75cc8d5cb7-5hcqw\" (UID: \"fa51c4cf-ab54-439e-a656-a7f2cfa06e6b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:37.478457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.478430 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9z8\" (UniqueName: \"kubernetes.io/projected/fa51c4cf-ab54-439e-a656-a7f2cfa06e6b-kube-api-access-9b9z8\") pod \"authorino-operator-75cc8d5cb7-5hcqw\" (UID: \"fa51c4cf-ab54-439e-a656-a7f2cfa06e6b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:37.643497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.643410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:37.778871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.778841 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw"] Jul 12 17:09:37.780782 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:37.780753 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa51c4cf_ab54_439e_a656_a7f2cfa06e6b.slice/crio-1f14534ceaf540e99d2ffafd7b7a6161913105a66ffa4599f85e78910d659a83 WatchSource:0}: Error finding container 1f14534ceaf540e99d2ffafd7b7a6161913105a66ffa4599f85e78910d659a83: Status 404 returned error can't find the container with id 1f14534ceaf540e99d2ffafd7b7a6161913105a66ffa4599f85e78910d659a83 Jul 12 17:09:37.874668 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.874635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" event={"ID":"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3","Type":"ContainerStarted","Data":"3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d"} Jul 12 17:09:37.875180 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.874813 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:37.875964 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.875938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" event={"ID":"fa51c4cf-ab54-439e-a656-a7f2cfa06e6b","Type":"ContainerStarted","Data":"1f14534ceaf540e99d2ffafd7b7a6161913105a66ffa4599f85e78910d659a83"} Jul 12 17:09:37.896318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:37.896219 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" podStartSLOduration=1.5175448519999999 podStartE2EDuration="7.896203537s" totalImagesPullingTime="6.378658685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:37.894112515 +0000 UTC m=+385.389018975" watchObservedRunningTime="2026-07-12 17:09:37.896203537 +0000 UTC m=+385.391109976" Jul 12 17:09:38.552246 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:38.552210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:38.646850 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:38.646816 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:38.819964 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:38.819863 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:38.894773 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:38.894723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:38.994701 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:38.994665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:40.474513 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:40.474481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:40.711795 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:40.711766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:40.894375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:40.894282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" event={"ID":"fa51c4cf-ab54-439e-a656-a7f2cfa06e6b","Type":"ContainerStarted","Data":"0268546519a89c77f1a3af4b6884cece673adeb3b881b2749d503691bd56f709"} Jul 12 17:09:40.894375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:40.894373 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:40.911120 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:40.911074 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" podStartSLOduration=1.665139908 podStartE2EDuration="3.911059708s" totalImagesPullingTime="2.2459198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:40.909134084 +0000 UTC m=+388.404040570" watchObservedRunningTime="2026-07-12 17:09:40.911059708 +0000 UTC m=+388.405966149" Jul 12 17:09:42.076862 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:42.076833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:42.138461 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:42.138431 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:47.878592 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:47.878558 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7nlfs" Jul 12 17:09:48.463798 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:48.463765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:48.569460 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:48.569430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:48.885081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:48.885009 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:49.991823 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:49.991794 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9"] Jul 12 17:09:49.995326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:49.995309 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.008536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.008513 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9"] Jul 12 17:09:50.089274 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.089238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.089422 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.089386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cclz\" (UniqueName: \"kubernetes.io/projected/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-kube-api-access-2cclz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.190487 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.190446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cclz\" (UniqueName: \"kubernetes.io/projected/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-kube-api-access-2cclz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.190733 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.190515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.190966 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.190897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.195107 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.195079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:50.201110 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.201071 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cclz\" (UniqueName: \"kubernetes.io/projected/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-kube-api-access-2cclz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.292546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.292472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:50.305461 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.305438 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.456953 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.456906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9"] Jul 12 17:09:50.459665 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:50.459637 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9fde53_9567_4b1e_b72c_5be5a5a8b930.slice/crio-7d7405b128fdadf65911558109d2d85f317f82930cc203a83787717fc49f6179 WatchSource:0}: Error finding container 7d7405b128fdadf65911558109d2d85f317f82930cc203a83787717fc49f6179: Status 404 returned error can't find the container with id 7d7405b128fdadf65911558109d2d85f317f82930cc203a83787717fc49f6179 Jul 12 17:09:50.631691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.631614 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9"] Jul 12 17:09:50.637983 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.637958 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9"] Jul 12 17:09:50.651523 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.651499 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc"] Jul 12 17:09:50.651709 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.651670 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" containerName="manager" containerID="cri-o://3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d" gracePeriod=2 Jul 12 17:09:50.660329 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.660298 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc"] Jul 12 17:09:50.663047 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663019 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg"] Jul 12 17:09:50.663709 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663688 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" containerName="manager" Jul 12 17:09:50.663789 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663711 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" containerName="manager" Jul 12 17:09:50.663789 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663743 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" containerName="manager" Jul 12 17:09:50.663789 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663752 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" containerName="manager" Jul 12 17:09:50.663881 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663850 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" containerName="manager" Jul 12 17:09:50.663881 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.663865 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" containerName="manager" Jul 12 17:09:50.667228 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.667205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.670632 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.670609 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t"] Jul 12 17:09:50.670788 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.670768 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" containerName="manager" containerID="cri-o://eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4" gracePeriod=2 Jul 12 17:09:50.680537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.680515 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv"] Jul 12 17:09:50.684308 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.684285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.684542 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.684510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg"] Jul 12 17:09:50.693287 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.693265 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t"] Jul 12 17:09:50.700368 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.700341 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv"] Jul 12 17:09:50.705184 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.705163 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2"] Jul 12 17:09:50.705702 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.705686 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" containerName="manager" Jul 12 17:09:50.705810 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.705704 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" containerName="manager" Jul 12 17:09:50.705810 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.705779 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" containerName="manager" Jul 12 17:09:50.709321 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.709302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:09:50.709407 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.709315 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.726179 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.726153 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2"] Jul 12 17:09:50.737072 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.737036 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.762922 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.762883 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zp58t\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.796192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.796163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnbq\" (UniqueName: \"kubernetes.io/projected/c4703f38-751e-4e89-bbba-6d831638565d-kube-api-access-nxnbq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dcbzg\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.796348 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.796215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4703f38-751e-4e89-bbba-6d831638565d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dcbzg\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.796348 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.796254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkxs\" (UniqueName: \"kubernetes.io/projected/1814b74e-9cd4-4d70-ac44-86f5614fc31e-kube-api-access-2nkxs\") pod \"limitador-operator-controller-manager-859d7b78bd-b4tp2\" (UID: \"1814b74e-9cd4-4d70-ac44-86f5614fc31e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:09:50.796348 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.796293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvplm\" (UniqueName: \"kubernetes.io/projected/52b5e701-a506-4056-a2bd-e0dae076f64f-kube-api-access-gvplm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgbcv\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.796502 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.796405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52b5e701-a506-4056-a2bd-e0dae076f64f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgbcv\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.897762 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.897679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvplm\" (UniqueName: \"kubernetes.io/projected/52b5e701-a506-4056-a2bd-e0dae076f64f-kube-api-access-gvplm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgbcv\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.897762 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.897756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52b5e701-a506-4056-a2bd-e0dae076f64f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgbcv\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.897977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.897808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnbq\" (UniqueName: \"kubernetes.io/projected/c4703f38-751e-4e89-bbba-6d831638565d-kube-api-access-nxnbq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dcbzg\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.897977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.897845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4703f38-751e-4e89-bbba-6d831638565d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dcbzg\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.897977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.897881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkxs\" (UniqueName: \"kubernetes.io/projected/1814b74e-9cd4-4d70-ac44-86f5614fc31e-kube-api-access-2nkxs\") pod \"limitador-operator-controller-manager-859d7b78bd-b4tp2\" (UID: \"1814b74e-9cd4-4d70-ac44-86f5614fc31e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:09:50.898286 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.898264 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52b5e701-a506-4056-a2bd-e0dae076f64f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgbcv\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.898355 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.898305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4703f38-751e-4e89-bbba-6d831638565d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dcbzg\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.906676 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.906653 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnbq\" (UniqueName: \"kubernetes.io/projected/c4703f38-751e-4e89-bbba-6d831638565d-kube-api-access-nxnbq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dcbzg\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:50.907299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.907278 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvplm\" (UniqueName: \"kubernetes.io/projected/52b5e701-a506-4056-a2bd-e0dae076f64f-kube-api-access-gvplm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgbcv\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:50.907800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.907782 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkxs\" (UniqueName: \"kubernetes.io/projected/1814b74e-9cd4-4d70-ac44-86f5614fc31e-kube-api-access-2nkxs\") pod \"limitador-operator-controller-manager-859d7b78bd-b4tp2\" (UID: \"1814b74e-9cd4-4d70-ac44-86f5614fc31e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:09:50.923410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.923393 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.925444 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.925429 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.925635 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.925618 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.927559 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.927537 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zp58t\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.929418 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.929398 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.934978 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.931620 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zp58t\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.944901 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.944877 2573 generic.go:356] "Generic (PLEG): container finished" podID="185f5e63-882b-422a-96e3-3d5c42abfcbf" containerID="eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4" exitCode=0 Jul 12 17:09:50.945010 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.944944 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.945010 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.944975 2573 scope.go:117] "RemoveContainer" containerID="eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4" Jul 12 17:09:50.946894 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.946874 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" container="manager" Jul 12 17:09:50.947244 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.947217 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.948314 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.948295 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" containerID="3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d" exitCode=0 Jul 12 17:09:50.948388 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.948349 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.949138 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.949111 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zp58t\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.950944 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.950904 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-24sgc\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-24sgc" Jul 12 17:09:50.952682 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.952661 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:50.954179 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.954163 2573 scope.go:117] "RemoveContainer" containerID="eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4" Jul 12 17:09:50.954452 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.954430 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4\": container with ID starting with eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4 not found: ID does not exist" containerID="eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4" Jul 12 17:09:50.954508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.954465 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4"} err="failed to get container status \"eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4\": rpc error: code = NotFound desc = could not find container \"eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4\": container with ID starting with eb2a58b483953b40164a60f81424c938da79c847314a0b0c331eb5fd3a3808c4 not found: ID does not exist" Jul 12 17:09:50.954508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.954484 2573 scope.go:117] "RemoveContainer" containerID="3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d" Jul 12 17:09:50.954603 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.954586 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zp58t\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zp58t" Jul 12 17:09:50.961892 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.961880 2573 scope.go:117] "RemoveContainer" containerID="3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d" Jul 12 17:09:50.962192 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:50.962170 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d\": container with ID starting with 3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d not found: ID does not exist" containerID="3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d" Jul 12 17:09:50.962243 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.962205 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d"} err="failed to get container status \"3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d\": rpc error: code = NotFound desc = could not find container \"3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d\": container with ID starting with 3f78f03844ce66e5724550f4668170de5d224beb414d5c0ba9cbbfa2eb8ed86d not found: ID does not exist" Jul 12 17:09:50.998990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.998964 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-797fz\" (UniqueName: \"kubernetes.io/projected/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-kube-api-access-797fz\") pod \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " Jul 12 17:09:50.999332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.999012 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzrns\" (UniqueName: \"kubernetes.io/projected/185f5e63-882b-422a-96e3-3d5c42abfcbf-kube-api-access-mzrns\") pod \"185f5e63-882b-422a-96e3-3d5c42abfcbf\" (UID: \"185f5e63-882b-422a-96e3-3d5c42abfcbf\") " Jul 12 17:09:50.999332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.999065 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-extensions-socket-volume\") pod \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\" (UID: \"a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3\") " Jul 12 17:09:50.999611 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:50.999589 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" (UID: "a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:09:51.000860 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.000839 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-kube-api-access-797fz" (OuterVolumeSpecName: "kube-api-access-797fz") pod "a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" (UID: "a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3"). InnerVolumeSpecName "kube-api-access-797fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:09:51.000860 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.000848 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185f5e63-882b-422a-96e3-3d5c42abfcbf-kube-api-access-mzrns" (OuterVolumeSpecName: "kube-api-access-mzrns") pod "185f5e63-882b-422a-96e3-3d5c42abfcbf" (UID: "185f5e63-882b-422a-96e3-3d5c42abfcbf"). InnerVolumeSpecName "kube-api-access-mzrns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:09:51.038759 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.038721 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="185f5e63-882b-422a-96e3-3d5c42abfcbf" path="/var/lib/kubelet/pods/185f5e63-882b-422a-96e3-3d5c42abfcbf/volumes" Jul 12 17:09:51.039128 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.039114 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3" path="/var/lib/kubelet/pods/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3/volumes" Jul 12 17:09:51.060090 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.060065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:51.068359 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.068336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:51.073695 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.073671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:09:51.100373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.100314 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrns\" (UniqueName: \"kubernetes.io/projected/185f5e63-882b-422a-96e3-3d5c42abfcbf-kube-api-access-mzrns\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:51.100373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.100341 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-extensions-socket-volume\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:51.100373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.100351 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797fz\" (UniqueName: \"kubernetes.io/projected/a2b3741c-5ea0-4b34-b992-57c3ae4dcbf3-kube-api-access-797fz\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:51.219078 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.217864 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg"] Jul 12 17:09:51.223629 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:51.223333 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4703f38_751e_4e89_bbba_6d831638565d.slice/crio-331aa022d7eb76e1b20d26a1e1f4e3322b2c10ced48f095da375a9afab7cb2ab WatchSource:0}: Error finding container 331aa022d7eb76e1b20d26a1e1f4e3322b2c10ced48f095da375a9afab7cb2ab: Status 404 returned error can't find the container with id 331aa022d7eb76e1b20d26a1e1f4e3322b2c10ced48f095da375a9afab7cb2ab Jul 12 17:09:51.249978 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.249953 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv"] Jul 12 17:09:51.256438 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:51.256405 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b5e701_a506_4056_a2bd_e0dae076f64f.slice/crio-6ee63595ae8ada7753903da1c82d004bbde9b825da430b0773b2bb5671869d68 WatchSource:0}: Error finding container 6ee63595ae8ada7753903da1c82d004bbde9b825da430b0773b2bb5671869d68: Status 404 returned error can't find the container with id 6ee63595ae8ada7753903da1c82d004bbde9b825da430b0773b2bb5671869d68 Jul 12 17:09:51.274388 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.274368 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2"] Jul 12 17:09:51.277130 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:09:51.277097 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1814b74e_9cd4_4d70_ac44_86f5614fc31e.slice/crio-923bdef98ecadaa0c73734a49dbe26f92f1099156344a2a9d4c140785c495158 WatchSource:0}: Error finding container 923bdef98ecadaa0c73734a49dbe26f92f1099156344a2a9d4c140785c495158: Status 404 returned error can't find the container with id 923bdef98ecadaa0c73734a49dbe26f92f1099156344a2a9d4c140785c495158 Jul 12 17:09:51.902360 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.902331 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5hcqw" Jul 12 17:09:51.957117 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.957077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" event={"ID":"1814b74e-9cd4-4d70-ac44-86f5614fc31e","Type":"ContainerStarted","Data":"b08eec37cd013c4f6224ddd8fa824731806335f88ab2f317053141ff28fdef70"} Jul 12 17:09:51.957299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.957144 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" event={"ID":"1814b74e-9cd4-4d70-ac44-86f5614fc31e","Type":"ContainerStarted","Data":"923bdef98ecadaa0c73734a49dbe26f92f1099156344a2a9d4c140785c495158"} Jul 12 17:09:51.957299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.957271 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:09:51.959838 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.959806 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" containerName="manager" containerID="cri-o://29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9" gracePeriod=2 Jul 12 17:09:51.961577 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.961554 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" event={"ID":"c4703f38-751e-4e89-bbba-6d831638565d","Type":"ContainerStarted","Data":"0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153"} Jul 12 17:09:51.961701 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.961596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" event={"ID":"c4703f38-751e-4e89-bbba-6d831638565d","Type":"ContainerStarted","Data":"331aa022d7eb76e1b20d26a1e1f4e3322b2c10ced48f095da375a9afab7cb2ab"} Jul 12 17:09:51.961763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.961717 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:09:51.963511 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.963481 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" event={"ID":"52b5e701-a506-4056-a2bd-e0dae076f64f","Type":"ContainerStarted","Data":"eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2"} Jul 12 17:09:51.963615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.963525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" event={"ID":"52b5e701-a506-4056-a2bd-e0dae076f64f","Type":"ContainerStarted","Data":"6ee63595ae8ada7753903da1c82d004bbde9b825da430b0773b2bb5671869d68"} Jul 12 17:09:51.963615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.963565 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:09:51.978435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.978396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" podStartSLOduration=1.97838263 podStartE2EDuration="1.97838263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:51.977749317 +0000 UTC m=+399.472655780" watchObservedRunningTime="2026-07-12 17:09:51.97838263 +0000 UTC m=+399.473289070" Jul 12 17:09:51.999442 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:51.999399 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" podStartSLOduration=1.999384914 podStartE2EDuration="1.999384914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:51.998220251 +0000 UTC m=+399.493126717" watchObservedRunningTime="2026-07-12 17:09:51.999384914 +0000 UTC m=+399.494291354" Jul 12 17:09:52.027067 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.027010 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" podStartSLOduration=2.026991406 podStartE2EDuration="2.026991406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:09:52.025825214 +0000 UTC m=+399.520731654" watchObservedRunningTime="2026-07-12 17:09:52.026991406 +0000 UTC m=+399.521897845" Jul 12 17:09:52.194430 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.194400 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:52.199950 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:52.199924 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:52.311175 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.311137 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-extensions-socket-volume\") pod \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " Jul 12 17:09:52.311315 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.311209 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cclz\" (UniqueName: \"kubernetes.io/projected/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-kube-api-access-2cclz\") pod \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\" (UID: \"8b9fde53-9567-4b1e-b72c-5be5a5a8b930\") " Jul 12 17:09:52.311378 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.311354 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8b9fde53-9567-4b1e-b72c-5be5a5a8b930" (UID: "8b9fde53-9567-4b1e-b72c-5be5a5a8b930"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:09:52.313194 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.313171 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-kube-api-access-2cclz" (OuterVolumeSpecName: "kube-api-access-2cclz") pod "8b9fde53-9567-4b1e-b72c-5be5a5a8b930" (UID: "8b9fde53-9567-4b1e-b72c-5be5a5a8b930"). InnerVolumeSpecName "kube-api-access-2cclz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:09:52.412574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.412543 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-extensions-socket-volume\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:52.412574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.412572 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cclz\" (UniqueName: \"kubernetes.io/projected/8b9fde53-9567-4b1e-b72c-5be5a5a8b930-kube-api-access-2cclz\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:52.971019 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.970988 2573 generic.go:356] "Generic (PLEG): container finished" podID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" containerID="29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9" exitCode=2 Jul 12 17:09:52.971174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.971038 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:52.971174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.971048 2573 scope.go:117] "RemoveContainer" containerID="29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9" Jul 12 17:09:52.973449 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:52.973423 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:52.980336 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.980322 2573 scope.go:117] "RemoveContainer" containerID="29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9" Jul 12 17:09:52.980581 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:52.980564 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9\": container with ID starting with 29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9 not found: ID does not exist" containerID="29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9" Jul 12 17:09:52.980642 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:52.980590 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9"} err="failed to get container status \"29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9\": rpc error: code = NotFound desc = could not find container \"29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9\": container with ID starting with 29d53e61e7d9ce5ce6798fc1fb27781cb30b8b041e933c8cddc5b2c5a04d74f9 not found: ID does not exist" Jul 12 17:09:52.981660 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:52.981639 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:53.037694 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:53.037661 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmtj9\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmtj9" Jul 12 17:09:53.039100 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:53.039081 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9fde53-9567-4b1e-b72c-5be5a5a8b930" path="/var/lib/kubelet/pods/8b9fde53-9567-4b1e-b72c-5be5a5a8b930/volumes" Jul 12 17:09:54.892301 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:54.892259 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:54.992602 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:54.992560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:55.908499 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:55.908459 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-756f7d7b75-hzwbf" podUID="d526e291-1569-4b11-a004-789f57269494" containerName="console" containerID="cri-o://614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df" gracePeriod=15 Jul 12 17:09:56.165933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.165854 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:09:56.249823 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.249799 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5sbh\" (UniqueName: \"kubernetes.io/projected/d526e291-1569-4b11-a004-789f57269494-kube-api-access-k5sbh\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.249997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.249842 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-oauth-config\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.249997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.249875 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-trusted-ca-bundle\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.249997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.249901 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-oauth-serving-cert\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.250223 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250018 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-serving-cert\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.250223 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250054 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-console-config\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.250223 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250100 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-service-ca\") pod \"d526e291-1569-4b11-a004-789f57269494\" (UID: \"d526e291-1569-4b11-a004-789f57269494\") " Jul 12 17:09:56.250379 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250351 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:09:56.250435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250358 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:09:56.250493 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250436 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-console-config" (OuterVolumeSpecName: "console-config") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:09:56.250575 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.250550 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-service-ca" (OuterVolumeSpecName: "service-ca") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:09:56.252097 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.252063 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:09:56.252219 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.252147 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:09:56.252219 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.252164 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d526e291-1569-4b11-a004-789f57269494-kube-api-access-k5sbh" (OuterVolumeSpecName: "kube-api-access-k5sbh") pod "d526e291-1569-4b11-a004-789f57269494" (UID: "d526e291-1569-4b11-a004-789f57269494"). InnerVolumeSpecName "kube-api-access-k5sbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:09:56.351178 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351144 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.351317 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351180 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5sbh\" (UniqueName: \"kubernetes.io/projected/d526e291-1569-4b11-a004-789f57269494-kube-api-access-k5sbh\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.351317 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351202 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-oauth-config\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.351317 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351211 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-trusted-ca-bundle\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.351317 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351220 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-oauth-serving-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.351317 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351228 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d526e291-1569-4b11-a004-789f57269494-console-serving-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.351317 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.351236 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d526e291-1569-4b11-a004-789f57269494-console-config\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:09:56.993596 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.993562 2573 generic.go:356] "Generic (PLEG): container finished" podID="d526e291-1569-4b11-a004-789f57269494" containerID="614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df" exitCode=0 Jul 12 17:09:56.993988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.993606 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f7d7b75-hzwbf" Jul 12 17:09:56.993988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.993645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f7d7b75-hzwbf" event={"ID":"d526e291-1569-4b11-a004-789f57269494","Type":"ContainerDied","Data":"614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df"} Jul 12 17:09:56.993988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.993697 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f7d7b75-hzwbf" event={"ID":"d526e291-1569-4b11-a004-789f57269494","Type":"ContainerDied","Data":"2a0419678d80946bd12588399a08afb9f1934682b10b7b4689db472e20cb33b1"} Jul 12 17:09:56.993988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:56.993719 2573 scope.go:117] "RemoveContainer" containerID="614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df" Jul 12 17:09:57.002636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.002620 2573 scope.go:117] "RemoveContainer" containerID="614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df" Jul 12 17:09:57.002922 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:09:57.002893 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df\": container with ID starting with 614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df not found: ID does not exist" containerID="614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df" Jul 12 17:09:57.002971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.002934 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df"} err="failed to get container status \"614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df\": rpc error: code = NotFound desc = could not find container \"614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df\": container with ID starting with 614ffc86c750bb01d13e1a8f538a20c54a8f057866596bfdb9f0d952b5cc23df not found: ID does not exist" Jul 12 17:09:57.018652 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.018627 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-756f7d7b75-hzwbf"] Jul 12 17:09:57.020343 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.020326 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-756f7d7b75-hzwbf"] Jul 12 17:09:57.038573 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.038552 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d526e291-1569-4b11-a004-789f57269494" path="/var/lib/kubelet/pods/d526e291-1569-4b11-a004-789f57269494/volumes" Jul 12 17:09:57.488624 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.488587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:09:57.588566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:09:57.588536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:00.288094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:00.288063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:00.347829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:00.347801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:02.973424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:02.973397 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:10:02.973837 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:02.973465 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:10:02.973837 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:02.973589 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4tp2" Jul 12 17:10:03.047275 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.047238 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv"] Jul 12 17:10:03.047473 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.047447 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" podUID="52b5e701-a506-4056-a2bd-e0dae076f64f" containerName="manager" containerID="cri-o://eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2" gracePeriod=10 Jul 12 17:10:03.293444 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.293425 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:10:03.325465 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.325427 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc"] Jul 12 17:10:03.325963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.325939 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b5e701-a506-4056-a2bd-e0dae076f64f" containerName="manager" Jul 12 17:10:03.325963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.325961 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b5e701-a506-4056-a2bd-e0dae076f64f" containerName="manager" Jul 12 17:10:03.326133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.325976 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d526e291-1569-4b11-a004-789f57269494" containerName="console" Jul 12 17:10:03.326133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.325984 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d526e291-1569-4b11-a004-789f57269494" containerName="console" Jul 12 17:10:03.326133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.326068 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b5e701-a506-4056-a2bd-e0dae076f64f" containerName="manager" Jul 12 17:10:03.326133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.326078 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d526e291-1569-4b11-a004-789f57269494" containerName="console" Jul 12 17:10:03.329614 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.329596 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.349719 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.349686 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc"] Jul 12 17:10:03.408508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.408476 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvplm\" (UniqueName: \"kubernetes.io/projected/52b5e701-a506-4056-a2bd-e0dae076f64f-kube-api-access-gvplm\") pod \"52b5e701-a506-4056-a2bd-e0dae076f64f\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " Jul 12 17:10:03.408662 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.408571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52b5e701-a506-4056-a2bd-e0dae076f64f-extensions-socket-volume\") pod \"52b5e701-a506-4056-a2bd-e0dae076f64f\" (UID: \"52b5e701-a506-4056-a2bd-e0dae076f64f\") " Jul 12 17:10:03.408957 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.408926 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b5e701-a506-4056-a2bd-e0dae076f64f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "52b5e701-a506-4056-a2bd-e0dae076f64f" (UID: "52b5e701-a506-4056-a2bd-e0dae076f64f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:10:03.410598 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.410579 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b5e701-a506-4056-a2bd-e0dae076f64f-kube-api-access-gvplm" (OuterVolumeSpecName: "kube-api-access-gvplm") pod "52b5e701-a506-4056-a2bd-e0dae076f64f" (UID: "52b5e701-a506-4056-a2bd-e0dae076f64f"). InnerVolumeSpecName "kube-api-access-gvplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:10:03.510124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.510055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q75bc\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.510124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.510099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpwv\" (UniqueName: \"kubernetes.io/projected/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-kube-api-access-ptpwv\") pod \"kuadrant-operator-controller-manager-86878c769f-q75bc\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.510290 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.510227 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvplm\" (UniqueName: \"kubernetes.io/projected/52b5e701-a506-4056-a2bd-e0dae076f64f-kube-api-access-gvplm\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:03.510290 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.510247 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52b5e701-a506-4056-a2bd-e0dae076f64f-extensions-socket-volume\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:03.611426 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.611390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q75bc\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.611566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.611443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpwv\" (UniqueName: \"kubernetes.io/projected/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-kube-api-access-ptpwv\") pod \"kuadrant-operator-controller-manager-86878c769f-q75bc\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.611765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.611744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q75bc\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.620294 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.620268 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpwv\" (UniqueName: \"kubernetes.io/projected/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-kube-api-access-ptpwv\") pod \"kuadrant-operator-controller-manager-86878c769f-q75bc\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.650251 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.650227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:03.783568 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:03.783538 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc"] Jul 12 17:10:03.786471 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:10:03.786442 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9b3a6d_0a70_4445_b43f_48fab5b5bd71.slice/crio-838f71621423b2b788a64389a1853479823aaace33802288ab77bfc35158f6d5 WatchSource:0}: Error finding container 838f71621423b2b788a64389a1853479823aaace33802288ab77bfc35158f6d5: Status 404 returned error can't find the container with id 838f71621423b2b788a64389a1853479823aaace33802288ab77bfc35158f6d5 Jul 12 17:10:04.034499 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.034385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" event={"ID":"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71","Type":"ContainerStarted","Data":"4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972"} Jul 12 17:10:04.034499 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.034450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" event={"ID":"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71","Type":"ContainerStarted","Data":"838f71621423b2b788a64389a1853479823aaace33802288ab77bfc35158f6d5"} Jul 12 17:10:04.035039 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.034611 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:04.035953 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.035903 2573 generic.go:356] "Generic (PLEG): container finished" podID="52b5e701-a506-4056-a2bd-e0dae076f64f" containerID="eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2" exitCode=0 Jul 12 17:10:04.036094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.035977 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" Jul 12 17:10:04.036094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.036016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" event={"ID":"52b5e701-a506-4056-a2bd-e0dae076f64f","Type":"ContainerDied","Data":"eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2"} Jul 12 17:10:04.036094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.036060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv" event={"ID":"52b5e701-a506-4056-a2bd-e0dae076f64f","Type":"ContainerDied","Data":"6ee63595ae8ada7753903da1c82d004bbde9b825da430b0773b2bb5671869d68"} Jul 12 17:10:04.036094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.036087 2573 scope.go:117] "RemoveContainer" containerID="eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2" Jul 12 17:10:04.045496 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.045476 2573 scope.go:117] "RemoveContainer" containerID="eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2" Jul 12 17:10:04.045819 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:10:04.045796 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2\": container with ID starting with eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2 not found: ID does not exist" containerID="eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2" Jul 12 17:10:04.045955 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.045824 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2"} err="failed to get container status \"eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2\": rpc error: code = NotFound desc = could not find container \"eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2\": container with ID starting with eb5a6a7f363e2a479e7aef10a871bc980a5a1f9937a3e969e162700f21c222b2 not found: ID does not exist" Jul 12 17:10:04.055204 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.055013 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" podStartSLOduration=1.054991562 podStartE2EDuration="1.054991562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:04.054205186 +0000 UTC m=+411.549111628" watchObservedRunningTime="2026-07-12 17:10:04.054991562 +0000 UTC m=+411.549898001" Jul 12 17:10:04.081402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.081370 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv"] Jul 12 17:10:04.090028 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:04.089998 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgbcv"] Jul 12 17:10:05.039434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:05.039401 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b5e701-a506-4056-a2bd-e0dae076f64f" path="/var/lib/kubelet/pods/52b5e701-a506-4056-a2bd-e0dae076f64f/volumes" Jul 12 17:10:07.288977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:07.288949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:07.389713 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:07.389678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:13.702212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:13.702175 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:13.812193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:13.812154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:15.044428 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.044398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:10:15.098158 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.098123 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg"] Jul 12 17:10:15.098346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.098310 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" podUID="c4703f38-751e-4e89-bbba-6d831638565d" containerName="manager" containerID="cri-o://0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153" gracePeriod=10 Jul 12 17:10:15.354551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.354528 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:10:15.411984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.411950 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxnbq\" (UniqueName: \"kubernetes.io/projected/c4703f38-751e-4e89-bbba-6d831638565d-kube-api-access-nxnbq\") pod \"c4703f38-751e-4e89-bbba-6d831638565d\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " Jul 12 17:10:15.412150 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.411995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4703f38-751e-4e89-bbba-6d831638565d-extensions-socket-volume\") pod \"c4703f38-751e-4e89-bbba-6d831638565d\" (UID: \"c4703f38-751e-4e89-bbba-6d831638565d\") " Jul 12 17:10:15.412363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.412339 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4703f38-751e-4e89-bbba-6d831638565d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c4703f38-751e-4e89-bbba-6d831638565d" (UID: "c4703f38-751e-4e89-bbba-6d831638565d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:10:15.413983 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.413958 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4703f38-751e-4e89-bbba-6d831638565d-kube-api-access-nxnbq" (OuterVolumeSpecName: "kube-api-access-nxnbq") pod "c4703f38-751e-4e89-bbba-6d831638565d" (UID: "c4703f38-751e-4e89-bbba-6d831638565d"). InnerVolumeSpecName "kube-api-access-nxnbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:10:15.513325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.513280 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnbq\" (UniqueName: \"kubernetes.io/projected/c4703f38-751e-4e89-bbba-6d831638565d-kube-api-access-nxnbq\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:15.513325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:15.513316 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4703f38-751e-4e89-bbba-6d831638565d-extensions-socket-volume\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:16.103882 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.103844 2573 generic.go:356] "Generic (PLEG): container finished" podID="c4703f38-751e-4e89-bbba-6d831638565d" containerID="0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153" exitCode=0 Jul 12 17:10:16.104268 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.103888 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" Jul 12 17:10:16.104268 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.103901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" event={"ID":"c4703f38-751e-4e89-bbba-6d831638565d","Type":"ContainerDied","Data":"0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153"} Jul 12 17:10:16.104268 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.103964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg" event={"ID":"c4703f38-751e-4e89-bbba-6d831638565d","Type":"ContainerDied","Data":"331aa022d7eb76e1b20d26a1e1f4e3322b2c10ced48f095da375a9afab7cb2ab"} Jul 12 17:10:16.104268 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.103986 2573 scope.go:117] "RemoveContainer" containerID="0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153" Jul 12 17:10:16.112758 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.112739 2573 scope.go:117] "RemoveContainer" containerID="0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153" Jul 12 17:10:16.113071 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:10:16.113052 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153\": container with ID starting with 0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153 not found: ID does not exist" containerID="0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153" Jul 12 17:10:16.113153 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.113080 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153"} err="failed to get container status \"0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153\": rpc error: code = NotFound desc = could not find container \"0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153\": container with ID starting with 0dbc2e08f6df045ae44999e1cdcd35cc716461e4dd7ef4e80abbfb970e3cd153 not found: ID does not exist" Jul 12 17:10:16.125951 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.125893 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg"] Jul 12 17:10:16.129880 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:16.129853 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dcbzg"] Jul 12 17:10:17.038680 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:17.038650 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4703f38-751e-4e89-bbba-6d831638565d" path="/var/lib/kubelet/pods/c4703f38-751e-4e89-bbba-6d831638565d/volumes" Jul 12 17:10:19.430471 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:19.430438 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:19.579816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:19.579787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:33.779139 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.779100 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:33.779630 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.779505 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4703f38-751e-4e89-bbba-6d831638565d" containerName="manager" Jul 12 17:10:33.779630 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.779518 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4703f38-751e-4e89-bbba-6d831638565d" containerName="manager" Jul 12 17:10:33.779630 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.779588 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4703f38-751e-4e89-bbba-6d831638565d" containerName="manager" Jul 12 17:10:33.784319 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.784294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:33.787132 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.787105 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 12 17:10:33.787132 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.787124 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2z6gm\"" Jul 12 17:10:33.788163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.788131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:33.791783 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.791754 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:33.880770 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.880735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:33.885303 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.885276 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-config-file\") pod \"limitador-limitador-64c94fd64f-fwx8f\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:33.885470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.885322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2kbl\" (UniqueName: \"kubernetes.io/projected/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-kube-api-access-b2kbl\") pod \"limitador-limitador-64c94fd64f-fwx8f\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:33.886375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.886344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:33.939588 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.939554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:33.986621 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.986586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-config-file\") pod \"limitador-limitador-64c94fd64f-fwx8f\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:33.986791 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.986636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2kbl\" (UniqueName: \"kubernetes.io/projected/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-kube-api-access-b2kbl\") pod \"limitador-limitador-64c94fd64f-fwx8f\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:33.987465 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:33.987443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-config-file\") pod \"limitador-limitador-64c94fd64f-fwx8f\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:34.000849 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.000814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2kbl\" (UniqueName: \"kubernetes.io/projected/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-kube-api-access-b2kbl\") pod \"limitador-limitador-64c94fd64f-fwx8f\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:34.096969 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.096856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:34.231619 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.231597 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:34.233821 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:10:34.233793 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cc5ed3_4d5e_4457_9a6a_6c79206c914a.slice/crio-de8f345c4ee96767ecb57180e20f24ae8f191e90e0c9968b5f61771ca21a4285 WatchSource:0}: Error finding container de8f345c4ee96767ecb57180e20f24ae8f191e90e0c9968b5f61771ca21a4285: Status 404 returned error can't find the container with id de8f345c4ee96767ecb57180e20f24ae8f191e90e0c9968b5f61771ca21a4285 Jul 12 17:10:34.579147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.579108 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-6j7gl"] Jul 12 17:10:34.584412 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.584388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:34.586785 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.586752 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j8ms5\"" Jul 12 17:10:34.589754 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.589726 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6j7gl"] Jul 12 17:10:34.607561 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.607524 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:34.664364 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.664275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:34.693574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.693532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwqk\" (UniqueName: \"kubernetes.io/projected/d2f958b3-5cd4-4b75-827a-816dbe5733a6-kube-api-access-brwqk\") pod \"authorino-58cd955675-6j7gl\" (UID: \"d2f958b3-5cd4-4b75-827a-816dbe5733a6\") " pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:34.794515 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.794466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwqk\" (UniqueName: \"kubernetes.io/projected/d2f958b3-5cd4-4b75-827a-816dbe5733a6-kube-api-access-brwqk\") pod \"authorino-58cd955675-6j7gl\" (UID: \"d2f958b3-5cd4-4b75-827a-816dbe5733a6\") " pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:34.802884 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.802849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwqk\" (UniqueName: \"kubernetes.io/projected/d2f958b3-5cd4-4b75-827a-816dbe5733a6-kube-api-access-brwqk\") pod \"authorino-58cd955675-6j7gl\" (UID: \"d2f958b3-5cd4-4b75-827a-816dbe5733a6\") " pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:34.896038 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:34.895955 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:35.068330 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:35.068300 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6j7gl"] Jul 12 17:10:35.070903 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:10:35.070868 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2f958b3_5cd4_4b75_827a_816dbe5733a6.slice/crio-f4893549174d8c2f8388691ccf35c533e5d75dcd904d07c0ee7976a80430e0bd WatchSource:0}: Error finding container f4893549174d8c2f8388691ccf35c533e5d75dcd904d07c0ee7976a80430e0bd: Status 404 returned error can't find the container with id f4893549174d8c2f8388691ccf35c533e5d75dcd904d07c0ee7976a80430e0bd Jul 12 17:10:35.208702 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:35.208662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" event={"ID":"92cc5ed3-4d5e-4457-9a6a-6c79206c914a","Type":"ContainerStarted","Data":"de8f345c4ee96767ecb57180e20f24ae8f191e90e0c9968b5f61771ca21a4285"} Jul 12 17:10:35.210396 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:35.210363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6j7gl" event={"ID":"d2f958b3-5cd4-4b75-827a-816dbe5733a6","Type":"ContainerStarted","Data":"f4893549174d8c2f8388691ccf35c533e5d75dcd904d07c0ee7976a80430e0bd"} Jul 12 17:10:39.233566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:39.233525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" event={"ID":"92cc5ed3-4d5e-4457-9a6a-6c79206c914a","Type":"ContainerStarted","Data":"f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609"} Jul 12 17:10:39.233998 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:39.233592 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:39.235037 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:39.235011 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6j7gl" event={"ID":"d2f958b3-5cd4-4b75-827a-816dbe5733a6","Type":"ContainerStarted","Data":"dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31"} Jul 12 17:10:39.251192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:39.251128 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" podStartSLOduration=1.613664609 podStartE2EDuration="6.251096193s" totalImagesPullingTime="4.637431584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:39.250020293 +0000 UTC m=+446.744926754" watchObservedRunningTime="2026-07-12 17:10:39.251096193 +0000 UTC m=+446.746002633" Jul 12 17:10:39.266217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:39.266163 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-6j7gl" podStartSLOduration=1.465823453 podStartE2EDuration="5.266146131s" totalImagesPullingTime="3.800322678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:39.265519044 +0000 UTC m=+446.760425484" watchObservedRunningTime="2026-07-12 17:10:39.266146131 +0000 UTC m=+446.761052570" Jul 12 17:10:39.380064 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:39.380022 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6j7gl"] Jul 12 17:10:41.246650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:41.246603 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-6j7gl" podUID="d2f958b3-5cd4-4b75-827a-816dbe5733a6" containerName="authorino" containerID="cri-o://dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31" gracePeriod=30 Jul 12 17:10:41.497705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:41.497647 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:41.659969 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:41.659893 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brwqk\" (UniqueName: \"kubernetes.io/projected/d2f958b3-5cd4-4b75-827a-816dbe5733a6-kube-api-access-brwqk\") pod \"d2f958b3-5cd4-4b75-827a-816dbe5733a6\" (UID: \"d2f958b3-5cd4-4b75-827a-816dbe5733a6\") " Jul 12 17:10:41.662043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:41.662017 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f958b3-5cd4-4b75-827a-816dbe5733a6-kube-api-access-brwqk" (OuterVolumeSpecName: "kube-api-access-brwqk") pod "d2f958b3-5cd4-4b75-827a-816dbe5733a6" (UID: "d2f958b3-5cd4-4b75-827a-816dbe5733a6"). InnerVolumeSpecName "kube-api-access-brwqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:10:41.761070 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:41.760990 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwqk\" (UniqueName: \"kubernetes.io/projected/d2f958b3-5cd4-4b75-827a-816dbe5733a6-kube-api-access-brwqk\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:42.252858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.252820 2573 generic.go:356] "Generic (PLEG): container finished" podID="d2f958b3-5cd4-4b75-827a-816dbe5733a6" containerID="dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31" exitCode=0 Jul 12 17:10:42.252858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.252858 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6j7gl" Jul 12 17:10:42.253325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.252863 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6j7gl" event={"ID":"d2f958b3-5cd4-4b75-827a-816dbe5733a6","Type":"ContainerDied","Data":"dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31"} Jul 12 17:10:42.253325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.252959 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6j7gl" event={"ID":"d2f958b3-5cd4-4b75-827a-816dbe5733a6","Type":"ContainerDied","Data":"f4893549174d8c2f8388691ccf35c533e5d75dcd904d07c0ee7976a80430e0bd"} Jul 12 17:10:42.253325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.252978 2573 scope.go:117] "RemoveContainer" containerID="dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31" Jul 12 17:10:42.261855 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.261837 2573 scope.go:117] "RemoveContainer" containerID="dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31" Jul 12 17:10:42.262174 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:10:42.262147 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31\": container with ID starting with dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31 not found: ID does not exist" containerID="dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31" Jul 12 17:10:42.262246 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.262176 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31"} err="failed to get container status \"dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31\": rpc error: code = NotFound desc = could not find container \"dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31\": container with ID starting with dd020a066006a6166a1f56e299f3e05baa211a839f5665f5a612bc3de753ce31 not found: ID does not exist" Jul 12 17:10:42.275208 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.275172 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6j7gl"] Jul 12 17:10:42.276970 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:42.276943 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-6j7gl"] Jul 12 17:10:43.044955 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:43.044899 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f958b3-5cd4-4b75-827a-816dbe5733a6" path="/var/lib/kubelet/pods/d2f958b3-5cd4-4b75-827a-816dbe5733a6/volumes" Jul 12 17:10:50.241705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:50.241675 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:50.389992 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:50.389951 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:50.390212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:50.390164 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" podUID="92cc5ed3-4d5e-4457-9a6a-6c79206c914a" containerName="limitador" containerID="cri-o://f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609" gracePeriod=30 Jul 12 17:10:50.966174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:50.966149 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:51.148806 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.148772 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2kbl\" (UniqueName: \"kubernetes.io/projected/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-kube-api-access-b2kbl\") pod \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " Jul 12 17:10:51.148806 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.148813 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-config-file\") pod \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\" (UID: \"92cc5ed3-4d5e-4457-9a6a-6c79206c914a\") " Jul 12 17:10:51.149296 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.149264 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-config-file" (OuterVolumeSpecName: "config-file") pod "92cc5ed3-4d5e-4457-9a6a-6c79206c914a" (UID: "92cc5ed3-4d5e-4457-9a6a-6c79206c914a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:10:51.150898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.150875 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-kube-api-access-b2kbl" (OuterVolumeSpecName: "kube-api-access-b2kbl") pod "92cc5ed3-4d5e-4457-9a6a-6c79206c914a" (UID: "92cc5ed3-4d5e-4457-9a6a-6c79206c914a"). InnerVolumeSpecName "kube-api-access-b2kbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:10:51.250485 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.250401 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2kbl\" (UniqueName: \"kubernetes.io/projected/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-kube-api-access-b2kbl\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:51.250485 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.250429 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92cc5ed3-4d5e-4457-9a6a-6c79206c914a-config-file\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:10:51.299779 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.299740 2573 generic.go:356] "Generic (PLEG): container finished" podID="92cc5ed3-4d5e-4457-9a6a-6c79206c914a" containerID="f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609" exitCode=0 Jul 12 17:10:51.299985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.299787 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" Jul 12 17:10:51.299985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.299833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" event={"ID":"92cc5ed3-4d5e-4457-9a6a-6c79206c914a","Type":"ContainerDied","Data":"f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609"} Jul 12 17:10:51.299985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.299884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fwx8f" event={"ID":"92cc5ed3-4d5e-4457-9a6a-6c79206c914a","Type":"ContainerDied","Data":"de8f345c4ee96767ecb57180e20f24ae8f191e90e0c9968b5f61771ca21a4285"} Jul 12 17:10:51.299985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.299948 2573 scope.go:117] "RemoveContainer" containerID="f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609" Jul 12 17:10:51.309606 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.309589 2573 scope.go:117] "RemoveContainer" containerID="f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609" Jul 12 17:10:51.309899 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:10:51.309880 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609\": container with ID starting with f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609 not found: ID does not exist" containerID="f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609" Jul 12 17:10:51.309986 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.309928 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609"} err="failed to get container status \"f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609\": rpc error: code = NotFound desc = could not find container \"f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609\": container with ID starting with f06517a37b7132f8039b8bd9e4bc830386b4e913578298aaadafb28185473609 not found: ID does not exist" Jul 12 17:10:51.321226 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.321193 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:51.325095 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:51.325071 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fwx8f"] Jul 12 17:10:53.039427 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.039396 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cc5ed3-4d5e-4457-9a6a-6c79206c914a" path="/var/lib/kubelet/pods/92cc5ed3-4d5e-4457-9a6a-6c79206c914a/volumes" Jul 12 17:10:53.131897 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.131855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:53.229742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.229706 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:53.648234 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648198 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bsczp"] Jul 12 17:10:53.648579 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648566 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cc5ed3-4d5e-4457-9a6a-6c79206c914a" containerName="limitador" Jul 12 17:10:53.648636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648580 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cc5ed3-4d5e-4457-9a6a-6c79206c914a" containerName="limitador" Jul 12 17:10:53.648636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648596 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f958b3-5cd4-4b75-827a-816dbe5733a6" containerName="authorino" Jul 12 17:10:53.648636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648601 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f958b3-5cd4-4b75-827a-816dbe5733a6" containerName="authorino" Jul 12 17:10:53.648727 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648668 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cc5ed3-4d5e-4457-9a6a-6c79206c914a" containerName="limitador" Jul 12 17:10:53.648727 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.648679 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f958b3-5cd4-4b75-827a-816dbe5733a6" containerName="authorino" Jul 12 17:10:53.652965 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.652946 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.657099 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.657075 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 12 17:10:53.657099 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.657075 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2bj9p\"" Jul 12 17:10:53.658253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.658219 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 12 17:10:53.658380 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.658329 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 12 17:10:53.660867 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.660843 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bsczp"] Jul 12 17:10:53.664044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.664021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:10:53.673165 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.673140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdp7h\" (UniqueName: \"kubernetes.io/projected/d56491b7-3ea2-46e7-8f57-4124f6767521-kube-api-access-gdp7h\") pod \"postgres-69fb4594cf-bsczp\" (UID: \"d56491b7-3ea2-46e7-8f57-4124f6767521\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.673262 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.673181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d56491b7-3ea2-46e7-8f57-4124f6767521-data\") pod \"postgres-69fb4594cf-bsczp\" (UID: \"d56491b7-3ea2-46e7-8f57-4124f6767521\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.773839 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.773806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdp7h\" (UniqueName: \"kubernetes.io/projected/d56491b7-3ea2-46e7-8f57-4124f6767521-kube-api-access-gdp7h\") pod \"postgres-69fb4594cf-bsczp\" (UID: \"d56491b7-3ea2-46e7-8f57-4124f6767521\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.774018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.773848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d56491b7-3ea2-46e7-8f57-4124f6767521-data\") pod \"postgres-69fb4594cf-bsczp\" (UID: \"d56491b7-3ea2-46e7-8f57-4124f6767521\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.774241 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.774221 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d56491b7-3ea2-46e7-8f57-4124f6767521-data\") pod \"postgres-69fb4594cf-bsczp\" (UID: \"d56491b7-3ea2-46e7-8f57-4124f6767521\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.784077 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.784054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdp7h\" (UniqueName: \"kubernetes.io/projected/d56491b7-3ea2-46e7-8f57-4124f6767521-kube-api-access-gdp7h\") pod \"postgres-69fb4594cf-bsczp\" (UID: \"d56491b7-3ea2-46e7-8f57-4124f6767521\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:53.963532 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:53.963498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:54.095678 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:54.095653 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bsczp"] Jul 12 17:10:54.097257 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:10:54.097225 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56491b7_3ea2_46e7_8f57_4124f6767521.slice/crio-f0d40d0d618116b65e713192d8b9059d140360373b1ab5b3815a2dd801959099 WatchSource:0}: Error finding container f0d40d0d618116b65e713192d8b9059d140360373b1ab5b3815a2dd801959099: Status 404 returned error can't find the container with id f0d40d0d618116b65e713192d8b9059d140360373b1ab5b3815a2dd801959099 Jul 12 17:10:54.317234 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:54.317139 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" event={"ID":"d56491b7-3ea2-46e7-8f57-4124f6767521","Type":"ContainerStarted","Data":"f0d40d0d618116b65e713192d8b9059d140360373b1ab5b3815a2dd801959099"} Jul 12 17:10:59.350944 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:59.350817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" event={"ID":"d56491b7-3ea2-46e7-8f57-4124f6767521","Type":"ContainerStarted","Data":"e603083e0fc0e93f489156438431f6a290123abe621619456c5c7b6e6235f52a"} Jul 12 17:10:59.351323 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:59.351056 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:10:59.366860 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:10:59.366815 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" podStartSLOduration=1.496062273 podStartE2EDuration="6.366801036s" totalImagesPullingTime="4.870738763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:59.365496859 +0000 UTC m=+466.860403300" watchObservedRunningTime="2026-07-12 17:10:59.366801036 +0000 UTC m=+466.861707475" Jul 12 17:11:05.387528 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.387491 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bsczp" Jul 12 17:11:05.710140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.710109 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:05.937720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.937684 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ds4j5"] Jul 12 17:11:05.941511 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.941485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:05.944171 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.944150 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j8ms5\"" Jul 12 17:11:05.952091 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.952060 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ds4j5"] Jul 12 17:11:05.990613 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.990535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gxlj\" (UniqueName: \"kubernetes.io/projected/9695ef69-3cc0-4bcf-b013-c810d76dd5fb-kube-api-access-9gxlj\") pod \"authorino-5d4b67c446-ds4j5\" (UID: \"9695ef69-3cc0-4bcf-b013-c810d76dd5fb\") " pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:05.991106 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:05.991082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:06.091354 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.091317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gxlj\" (UniqueName: \"kubernetes.io/projected/9695ef69-3cc0-4bcf-b013-c810d76dd5fb-kube-api-access-9gxlj\") pod \"authorino-5d4b67c446-ds4j5\" (UID: \"9695ef69-3cc0-4bcf-b013-c810d76dd5fb\") " pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:06.101766 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.101736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gxlj\" (UniqueName: \"kubernetes.io/projected/9695ef69-3cc0-4bcf-b013-c810d76dd5fb-kube-api-access-9gxlj\") pod \"authorino-5d4b67c446-ds4j5\" (UID: \"9695ef69-3cc0-4bcf-b013-c810d76dd5fb\") " pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:06.122093 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.122061 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ds4j5"] Jul 12 17:11:06.122337 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.122327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:06.150239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.150207 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-vthdn"] Jul 12 17:11:06.155585 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.155529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.159054 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.158992 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 12 17:11:06.160906 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.160852 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-vthdn"] Jul 12 17:11:06.193241 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.193202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stsv9\" (UniqueName: \"kubernetes.io/projected/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-kube-api-access-stsv9\") pod \"authorino-76877994-vthdn\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.193461 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.193286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-tls-cert\") pod \"authorino-76877994-vthdn\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.211820 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.211786 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-vthdn"] Jul 12 17:11:06.212118 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:11:06.212099 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-stsv9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-vthdn" podUID="67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf" Jul 12 17:11:06.232718 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.232559 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f584b7d89-7kx6f"] Jul 12 17:11:06.240255 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.240226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.243683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.243370 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f584b7d89-7kx6f"] Jul 12 17:11:06.275301 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.275275 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ds4j5"] Jul 12 17:11:06.277857 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:06.277814 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9695ef69_3cc0_4bcf_b013_c810d76dd5fb.slice/crio-bb1dfcba4629ac51e665b7f3cc8fe68bce52319d4d18782353caa595a8a6647a WatchSource:0}: Error finding container bb1dfcba4629ac51e665b7f3cc8fe68bce52319d4d18782353caa595a8a6647a: Status 404 returned error can't find the container with id bb1dfcba4629ac51e665b7f3cc8fe68bce52319d4d18782353caa595a8a6647a Jul 12 17:11:06.294422 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.294392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stsv9\" (UniqueName: \"kubernetes.io/projected/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-kube-api-access-stsv9\") pod \"authorino-76877994-vthdn\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.294562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.294448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-tls-cert\") pod \"authorino-76877994-vthdn\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.294562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.294499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-tls-cert\") pod \"authorino-f584b7d89-7kx6f\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.294849 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.294590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpt54\" (UniqueName: \"kubernetes.io/projected/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-kube-api-access-kpt54\") pod \"authorino-f584b7d89-7kx6f\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.297155 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.297133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-tls-cert\") pod \"authorino-76877994-vthdn\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.303234 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.303211 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stsv9\" (UniqueName: \"kubernetes.io/projected/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-kube-api-access-stsv9\") pod \"authorino-76877994-vthdn\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.388589 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.388549 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" event={"ID":"9695ef69-3cc0-4bcf-b013-c810d76dd5fb","Type":"ContainerStarted","Data":"bb1dfcba4629ac51e665b7f3cc8fe68bce52319d4d18782353caa595a8a6647a"} Jul 12 17:11:06.389050 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.388643 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.395128 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.395105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-tls-cert\") pod \"authorino-f584b7d89-7kx6f\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.395269 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.395179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpt54\" (UniqueName: \"kubernetes.io/projected/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-kube-api-access-kpt54\") pod \"authorino-f584b7d89-7kx6f\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.397645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.397621 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-tls-cert\") pod \"authorino-f584b7d89-7kx6f\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.406076 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.406052 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpt54\" (UniqueName: \"kubernetes.io/projected/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-kube-api-access-kpt54\") pod \"authorino-f584b7d89-7kx6f\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.410157 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.410135 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:06.496152 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.496057 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-tls-cert\") pod \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " Jul 12 17:11:06.496334 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.496186 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stsv9\" (UniqueName: \"kubernetes.io/projected/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-kube-api-access-stsv9\") pod \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\" (UID: \"67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf\") " Jul 12 17:11:06.498230 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.498207 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-kube-api-access-stsv9" (OuterVolumeSpecName: "kube-api-access-stsv9") pod "67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf" (UID: "67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf"). InnerVolumeSpecName "kube-api-access-stsv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:11:06.498305 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.498227 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf" (UID: "67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:11:06.553417 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.553382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:11:06.596971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.596938 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stsv9\" (UniqueName: \"kubernetes.io/projected/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-kube-api-access-stsv9\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:11:06.596971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.596969 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf-tls-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:11:06.689010 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:06.688986 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f584b7d89-7kx6f"] Jul 12 17:11:06.690814 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:06.690792 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8ef18c_4f55_4d42_909c_5e9ee29f6506.slice/crio-3a17f30bc33850e68dea3b7c877c86150e8c0ee5a8ba71fdb378c494c3f90a95 WatchSource:0}: Error finding container 3a17f30bc33850e68dea3b7c877c86150e8c0ee5a8ba71fdb378c494c3f90a95: Status 404 returned error can't find the container with id 3a17f30bc33850e68dea3b7c877c86150e8c0ee5a8ba71fdb378c494c3f90a95 Jul 12 17:11:07.396796 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.396699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f584b7d89-7kx6f" event={"ID":"bd8ef18c-4f55-4d42-909c-5e9ee29f6506","Type":"ContainerStarted","Data":"e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e"} Jul 12 17:11:07.396796 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.396753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f584b7d89-7kx6f" event={"ID":"bd8ef18c-4f55-4d42-909c-5e9ee29f6506","Type":"ContainerStarted","Data":"3a17f30bc33850e68dea3b7c877c86150e8c0ee5a8ba71fdb378c494c3f90a95"} Jul 12 17:11:07.398517 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.398489 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" event={"ID":"9695ef69-3cc0-4bcf-b013-c810d76dd5fb","Type":"ContainerStarted","Data":"b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff"} Jul 12 17:11:07.398644 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.398518 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" podUID="9695ef69-3cc0-4bcf-b013-c810d76dd5fb" containerName="authorino" containerID="cri-o://b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff" gracePeriod=30 Jul 12 17:11:07.398644 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.398534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vthdn" Jul 12 17:11:07.413926 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.413866 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f584b7d89-7kx6f" podStartSLOduration=0.897919596 podStartE2EDuration="1.413851302s" totalImagesPullingTime="515.931706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:07.412927858 +0000 UTC m=+474.907834315" watchObservedRunningTime="2026-07-12 17:11:07.413851302 +0000 UTC m=+474.908757741" Jul 12 17:11:07.444694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.439963 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-vthdn"] Jul 12 17:11:07.450363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.450332 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-vthdn"] Jul 12 17:11:07.466896 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.466833 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" podStartSLOduration=1.894886272 podStartE2EDuration="2.466812371s" totalImagesPullingTime="571.926099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:07.466331484 +0000 UTC m=+474.961237942" watchObservedRunningTime="2026-07-12 17:11:07.466812371 +0000 UTC m=+474.961718813" Jul 12 17:11:07.685769 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.685740 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:07.807210 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.807172 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gxlj\" (UniqueName: \"kubernetes.io/projected/9695ef69-3cc0-4bcf-b013-c810d76dd5fb-kube-api-access-9gxlj\") pod \"9695ef69-3cc0-4bcf-b013-c810d76dd5fb\" (UID: \"9695ef69-3cc0-4bcf-b013-c810d76dd5fb\") " Jul 12 17:11:07.809331 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.809296 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9695ef69-3cc0-4bcf-b013-c810d76dd5fb-kube-api-access-9gxlj" (OuterVolumeSpecName: "kube-api-access-9gxlj") pod "9695ef69-3cc0-4bcf-b013-c810d76dd5fb" (UID: "9695ef69-3cc0-4bcf-b013-c810d76dd5fb"). InnerVolumeSpecName "kube-api-access-9gxlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:11:07.908945 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:07.908901 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gxlj\" (UniqueName: \"kubernetes.io/projected/9695ef69-3cc0-4bcf-b013-c810d76dd5fb-kube-api-access-9gxlj\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:11:08.405670 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.405632 2573 generic.go:356] "Generic (PLEG): container finished" podID="9695ef69-3cc0-4bcf-b013-c810d76dd5fb" containerID="b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff" exitCode=0 Jul 12 17:11:08.406085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.405681 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" Jul 12 17:11:08.406085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.405722 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" event={"ID":"9695ef69-3cc0-4bcf-b013-c810d76dd5fb","Type":"ContainerDied","Data":"b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff"} Jul 12 17:11:08.406085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.405767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ds4j5" event={"ID":"9695ef69-3cc0-4bcf-b013-c810d76dd5fb","Type":"ContainerDied","Data":"bb1dfcba4629ac51e665b7f3cc8fe68bce52319d4d18782353caa595a8a6647a"} Jul 12 17:11:08.406085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.405787 2573 scope.go:117] "RemoveContainer" containerID="b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff" Jul 12 17:11:08.417393 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.417374 2573 scope.go:117] "RemoveContainer" containerID="b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff" Jul 12 17:11:08.417671 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:11:08.417646 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff\": container with ID starting with b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff not found: ID does not exist" containerID="b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff" Jul 12 17:11:08.417738 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.417681 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff"} err="failed to get container status \"b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff\": rpc error: code = NotFound desc = could not find container \"b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff\": container with ID starting with b46c1abf13655d41009bb6a0ba1d25e4f203b61a7bfed98e35e8667b4acea1ff not found: ID does not exist" Jul 12 17:11:08.427447 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.427075 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ds4j5"] Jul 12 17:11:08.429350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:08.429329 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ds4j5"] Jul 12 17:11:09.039793 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:09.039763 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf" path="/var/lib/kubelet/pods/67db3bc6-266f-4ef3-b2ca-8ba0d94d40bf/volumes" Jul 12 17:11:09.040015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:09.040002 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9695ef69-3cc0-4bcf-b013-c810d76dd5fb" path="/var/lib/kubelet/pods/9695ef69-3cc0-4bcf-b013-c810d76dd5fb/volumes" Jul 12 17:11:12.080357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.080316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:12.092674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.092642 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74f888f8cb-mxl5c"] Jul 12 17:11:12.093066 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.093053 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9695ef69-3cc0-4bcf-b013-c810d76dd5fb" containerName="authorino" Jul 12 17:11:12.093122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.093068 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9695ef69-3cc0-4bcf-b013-c810d76dd5fb" containerName="authorino" Jul 12 17:11:12.093163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.093139 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9695ef69-3cc0-4bcf-b013-c810d76dd5fb" containerName="authorino" Jul 12 17:11:12.116659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.116615 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f888f8cb-mxl5c"] Jul 12 17:11:12.116848 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.116756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.119695 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.119662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 12 17:11:12.119841 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.119794 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 17:11:12.119944 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.119794 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hx9r5\"" Jul 12 17:11:12.180817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.180780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:12.250761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.250728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6a2b378-eb5f-4854-b6a6-912b68e35133-webhook-cert\") pod \"maas-controller-74f888f8cb-mxl5c\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.250972 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.250769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5mr\" (UniqueName: \"kubernetes.io/projected/d6a2b378-eb5f-4854-b6a6-912b68e35133-kube-api-access-8v5mr\") pod \"maas-controller-74f888f8cb-mxl5c\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.272009 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.271958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:12.276532 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.276504 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-h9zds"] Jul 12 17:11:12.280252 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.280228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.288814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.288785 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-h9zds"] Jul 12 17:11:12.328214 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.328178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:12.351623 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.351533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6a2b378-eb5f-4854-b6a6-912b68e35133-webhook-cert\") pod \"maas-controller-74f888f8cb-mxl5c\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.351623 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.351578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5mr\" (UniqueName: \"kubernetes.io/projected/d6a2b378-eb5f-4854-b6a6-912b68e35133-kube-api-access-8v5mr\") pod \"maas-controller-74f888f8cb-mxl5c\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.351814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.351648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-webhook-cert\") pod \"maas-controller-5b576ccbb6-h9zds\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.351814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.351687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlgg\" (UniqueName: \"kubernetes.io/projected/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-kube-api-access-pqlgg\") pod \"maas-controller-5b576ccbb6-h9zds\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.354023 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.354002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6a2b378-eb5f-4854-b6a6-912b68e35133-webhook-cert\") pod \"maas-controller-74f888f8cb-mxl5c\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.360971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.360945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5mr\" (UniqueName: \"kubernetes.io/projected/d6a2b378-eb5f-4854-b6a6-912b68e35133-kube-api-access-8v5mr\") pod \"maas-controller-74f888f8cb-mxl5c\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.427390 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.427354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:12.452193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.452165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-webhook-cert\") pod \"maas-controller-5b576ccbb6-h9zds\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.452339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.452215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlgg\" (UniqueName: \"kubernetes.io/projected/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-kube-api-access-pqlgg\") pod \"maas-controller-5b576ccbb6-h9zds\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.454781 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.454759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-webhook-cert\") pod \"maas-controller-5b576ccbb6-h9zds\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.461204 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.461181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlgg\" (UniqueName: \"kubernetes.io/projected/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-kube-api-access-pqlgg\") pod \"maas-controller-5b576ccbb6-h9zds\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.566089 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.566052 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f888f8cb-mxl5c"] Jul 12 17:11:12.569130 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:12.569104 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a2b378_eb5f_4854_b6a6_912b68e35133.slice/crio-393585421c635cfb9eae17c3086b0eade760f16f72416283ffc83e6981150b0f WatchSource:0}: Error finding container 393585421c635cfb9eae17c3086b0eade760f16f72416283ffc83e6981150b0f: Status 404 returned error can't find the container with id 393585421c635cfb9eae17c3086b0eade760f16f72416283ffc83e6981150b0f Jul 12 17:11:12.591998 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.591967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:12.723735 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:12.723711 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-h9zds"] Jul 12 17:11:12.725510 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:12.725482 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee1b1d4_c9fe_46ae_adf3_f852dcf306da.slice/crio-2c0a9b6f169adb124d8b1626253723f593bb002ef95f4997f7857362ba3e94f1 WatchSource:0}: Error finding container 2c0a9b6f169adb124d8b1626253723f593bb002ef95f4997f7857362ba3e94f1: Status 404 returned error can't find the container with id 2c0a9b6f169adb124d8b1626253723f593bb002ef95f4997f7857362ba3e94f1 Jul 12 17:11:13.439192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:13.439140 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" event={"ID":"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da","Type":"ContainerStarted","Data":"2c0a9b6f169adb124d8b1626253723f593bb002ef95f4997f7857362ba3e94f1"} Jul 12 17:11:13.442002 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:13.441962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" event={"ID":"d6a2b378-eb5f-4854-b6a6-912b68e35133","Type":"ContainerStarted","Data":"393585421c635cfb9eae17c3086b0eade760f16f72416283ffc83e6981150b0f"} Jul 12 17:11:14.606784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:14.606756 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 17:11:14.813773 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:14.813744 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:14.914556 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:14.914521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:15.454976 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:15.454939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" event={"ID":"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da","Type":"ContainerStarted","Data":"e3ae08fb481706fa94d6354f7d2b694e4d7ee80c1b83ff44b0e1b1cfbd2201cd"} Jul 12 17:11:15.455181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:15.455005 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:15.456700 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:15.456666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" event={"ID":"d6a2b378-eb5f-4854-b6a6-912b68e35133","Type":"ContainerStarted","Data":"557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4"} Jul 12 17:11:15.456840 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:15.456731 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:15.472098 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:15.472024 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" podStartSLOduration=1.586945306 podStartE2EDuration="3.471988481s" totalImagesPullingTime="1.885043175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:15.471657951 +0000 UTC m=+482.966564415" watchObservedRunningTime="2026-07-12 17:11:15.471988481 +0000 UTC m=+482.966894923" Jul 12 17:11:15.492683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:15.492623 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" podStartSLOduration=1.459357988 podStartE2EDuration="3.492603889s" totalImagesPullingTime="2.033245901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:15.491250215 +0000 UTC m=+482.986156656" watchObservedRunningTime="2026-07-12 17:11:15.492603889 +0000 UTC m=+482.987510328" Jul 12 17:11:26.464386 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.464353 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:26.464772 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.464434 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:11:26.524262 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.524206 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74f888f8cb-mxl5c"] Jul 12 17:11:26.525046 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.524986 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" podUID="d6a2b378-eb5f-4854-b6a6-912b68e35133" containerName="manager" containerID="cri-o://557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4" gracePeriod=30 Jul 12 17:11:26.771518 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.771488 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:26.885535 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.885501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v5mr\" (UniqueName: \"kubernetes.io/projected/d6a2b378-eb5f-4854-b6a6-912b68e35133-kube-api-access-8v5mr\") pod \"d6a2b378-eb5f-4854-b6a6-912b68e35133\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " Jul 12 17:11:26.885745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.885571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6a2b378-eb5f-4854-b6a6-912b68e35133-webhook-cert\") pod \"d6a2b378-eb5f-4854-b6a6-912b68e35133\" (UID: \"d6a2b378-eb5f-4854-b6a6-912b68e35133\") " Jul 12 17:11:26.887890 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.887851 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a2b378-eb5f-4854-b6a6-912b68e35133-kube-api-access-8v5mr" (OuterVolumeSpecName: "kube-api-access-8v5mr") pod "d6a2b378-eb5f-4854-b6a6-912b68e35133" (UID: "d6a2b378-eb5f-4854-b6a6-912b68e35133"). InnerVolumeSpecName "kube-api-access-8v5mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:11:26.887890 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.887868 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a2b378-eb5f-4854-b6a6-912b68e35133-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d6a2b378-eb5f-4854-b6a6-912b68e35133" (UID: "d6a2b378-eb5f-4854-b6a6-912b68e35133"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:11:26.986482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.986432 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v5mr\" (UniqueName: \"kubernetes.io/projected/d6a2b378-eb5f-4854-b6a6-912b68e35133-kube-api-access-8v5mr\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:11:26.986482 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:26.986471 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6a2b378-eb5f-4854-b6a6-912b68e35133-webhook-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:11:27.519152 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.519113 2573 generic.go:356] "Generic (PLEG): container finished" podID="d6a2b378-eb5f-4854-b6a6-912b68e35133" containerID="557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4" exitCode=0 Jul 12 17:11:27.519574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.519168 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" Jul 12 17:11:27.519574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.519171 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" event={"ID":"d6a2b378-eb5f-4854-b6a6-912b68e35133","Type":"ContainerDied","Data":"557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4"} Jul 12 17:11:27.519574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.519279 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f888f8cb-mxl5c" event={"ID":"d6a2b378-eb5f-4854-b6a6-912b68e35133","Type":"ContainerDied","Data":"393585421c635cfb9eae17c3086b0eade760f16f72416283ffc83e6981150b0f"} Jul 12 17:11:27.519574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.519306 2573 scope.go:117] "RemoveContainer" containerID="557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4" Jul 12 17:11:27.528262 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.528079 2573 scope.go:117] "RemoveContainer" containerID="557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4" Jul 12 17:11:27.528377 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:11:27.528357 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4\": container with ID starting with 557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4 not found: ID does not exist" containerID="557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4" Jul 12 17:11:27.528418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.528385 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4"} err="failed to get container status \"557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4\": rpc error: code = NotFound desc = could not find container \"557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4\": container with ID starting with 557d8ce3b124b5035dbbf141ee2e915069f3fdd361a102bdba3169b277969fa4 not found: ID does not exist" Jul 12 17:11:27.535647 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.535621 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74f888f8cb-mxl5c"] Jul 12 17:11:27.539464 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:27.539443 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74f888f8cb-mxl5c"] Jul 12 17:11:29.039892 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:29.039857 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a2b378-eb5f-4854-b6a6-912b68e35133" path="/var/lib/kubelet/pods/d6a2b378-eb5f-4854-b6a6-912b68e35133/volumes" Jul 12 17:11:47.105347 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.105307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:47.179359 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.179325 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:47.555198 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.555163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:47.559568 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.559542 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj"] Jul 12 17:11:47.559964 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.559950 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a2b378-eb5f-4854-b6a6-912b68e35133" containerName="manager" Jul 12 17:11:47.560019 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.559966 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a2b378-eb5f-4854-b6a6-912b68e35133" containerName="manager" Jul 12 17:11:47.560055 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.560040 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a2b378-eb5f-4854-b6a6-912b68e35133" containerName="manager" Jul 12 17:11:47.563414 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.563395 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.566355 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.566338 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 12 17:11:47.566428 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.566353 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bljs8\"" Jul 12 17:11:47.574541 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.574515 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj"] Jul 12 17:11:47.652218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.652178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzv5l\" (UniqueName: \"kubernetes.io/projected/7acea634-f4c9-411a-a3e6-be310b9ac0bb-kube-api-access-lzv5l\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.652411 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.652225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acea634-f4c9-411a-a3e6-be310b9ac0bb-maas-api-tls\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.652411 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.652309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acea634-f4c9-411a-a3e6-be310b9ac0bb-openshift-service-ca\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.753029 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.752986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzv5l\" (UniqueName: \"kubernetes.io/projected/7acea634-f4c9-411a-a3e6-be310b9ac0bb-kube-api-access-lzv5l\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.753214 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.753038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acea634-f4c9-411a-a3e6-be310b9ac0bb-maas-api-tls\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.753214 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.753085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acea634-f4c9-411a-a3e6-be310b9ac0bb-openshift-service-ca\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.753214 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:11:47.753198 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 12 17:11:47.753404 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:11:47.753268 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acea634-f4c9-411a-a3e6-be310b9ac0bb-maas-api-tls podName:7acea634-f4c9-411a-a3e6-be310b9ac0bb nodeName:}" failed. No retries permitted until 2026-07-12 17:11:48.253252217 +0000 UTC m=+515.748158636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7acea634-f4c9-411a-a3e6-be310b9ac0bb-maas-api-tls") pod "maas-api-6db56bb667-z6tkj" (UID: "7acea634-f4c9-411a-a3e6-be310b9ac0bb") : secret "maas-api-serving-cert" not found Jul 12 17:11:47.753839 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.753815 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acea634-f4c9-411a-a3e6-be310b9ac0bb-openshift-service-ca\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.762189 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.762163 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzv5l\" (UniqueName: \"kubernetes.io/projected/7acea634-f4c9-411a-a3e6-be310b9ac0bb-kube-api-access-lzv5l\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:47.922894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:47.922855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:48.025280 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.025240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:48.093832 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.093801 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-k9cws"] Jul 12 17:11:48.100200 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.100177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.104159 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.104136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:48.105878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.105855 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xnv7x\"" Jul 12 17:11:48.106336 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.106322 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 12 17:11:48.112230 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.112206 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-k9cws"] Jul 12 17:11:48.140161 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.139883 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-4qp9w"] Jul 12 17:11:48.145645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.145616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.145792 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.145670 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-4qp9w"] Jul 12 17:11:48.157835 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.157806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-k9cws\" (UID: \"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.157994 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.157973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79d44\" (UniqueName: \"kubernetes.io/projected/f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a-kube-api-access-79d44\") pod \"payload-pre-processing-74584bb9f8-k9cws\" (UID: \"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.258695 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.258594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68tb\" (UniqueName: \"kubernetes.io/projected/a82263e2-db57-4511-9537-7d212c8bf5ea-kube-api-access-s68tb\") pod \"payload-processing-f9b8d9b9b-4qp9w\" (UID: \"a82263e2-db57-4511-9537-7d212c8bf5ea\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.258852 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.258721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-k9cws\" (UID: \"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.258852 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.258812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acea634-f4c9-411a-a3e6-be310b9ac0bb-maas-api-tls\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:48.258852 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.258843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79d44\" (UniqueName: \"kubernetes.io/projected/f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a-kube-api-access-79d44\") pod \"payload-pre-processing-74584bb9f8-k9cws\" (UID: \"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.259050 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.258873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82263e2-db57-4511-9537-7d212c8bf5ea-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-4qp9w\" (UID: \"a82263e2-db57-4511-9537-7d212c8bf5ea\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.259542 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.259517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-k9cws\" (UID: \"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.261337 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.261312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acea634-f4c9-411a-a3e6-be310b9ac0bb-maas-api-tls\") pod \"maas-api-6db56bb667-z6tkj\" (UID: \"7acea634-f4c9-411a-a3e6-be310b9ac0bb\") " pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:48.269024 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.268998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79d44\" (UniqueName: \"kubernetes.io/projected/f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a-kube-api-access-79d44\") pod \"payload-pre-processing-74584bb9f8-k9cws\" (UID: \"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.359859 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.359821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82263e2-db57-4511-9537-7d212c8bf5ea-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-4qp9w\" (UID: \"a82263e2-db57-4511-9537-7d212c8bf5ea\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.359859 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.359872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68tb\" (UniqueName: \"kubernetes.io/projected/a82263e2-db57-4511-9537-7d212c8bf5ea-kube-api-access-s68tb\") pod \"payload-processing-f9b8d9b9b-4qp9w\" (UID: \"a82263e2-db57-4511-9537-7d212c8bf5ea\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.360581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.360559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82263e2-db57-4511-9537-7d212c8bf5ea-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-4qp9w\" (UID: \"a82263e2-db57-4511-9537-7d212c8bf5ea\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.368476 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.368443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68tb\" (UniqueName: \"kubernetes.io/projected/a82263e2-db57-4511-9537-7d212c8bf5ea-kube-api-access-s68tb\") pod \"payload-processing-f9b8d9b9b-4qp9w\" (UID: \"a82263e2-db57-4511-9537-7d212c8bf5ea\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.412149 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.412109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:48.458213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.458158 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:48.474636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.474602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:48.576968 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.576933 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-k9cws"] Jul 12 17:11:48.581789 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:48.581755 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60ac4cb_9e42_41ff_b1bf_e7e72fccf66a.slice/crio-5a5d5d8f58027d11623fa86d9ccd738c8dc5b2a98d3fcca565eb1eb99eee20cb WatchSource:0}: Error finding container 5a5d5d8f58027d11623fa86d9ccd738c8dc5b2a98d3fcca565eb1eb99eee20cb: Status 404 returned error can't find the container with id 5a5d5d8f58027d11623fa86d9ccd738c8dc5b2a98d3fcca565eb1eb99eee20cb Jul 12 17:11:48.631772 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.631736 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" event={"ID":"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a","Type":"ContainerStarted","Data":"5a5d5d8f58027d11623fa86d9ccd738c8dc5b2a98d3fcca565eb1eb99eee20cb"} Jul 12 17:11:48.641703 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.641670 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-4qp9w"] Jul 12 17:11:48.645419 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:48.645387 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82263e2_db57_4511_9537_7d212c8bf5ea.slice/crio-adffef316650407aa89140d887520a81e6db46556915bc2e6dda7920074a3074 WatchSource:0}: Error finding container adffef316650407aa89140d887520a81e6db46556915bc2e6dda7920074a3074: Status 404 returned error can't find the container with id adffef316650407aa89140d887520a81e6db46556915bc2e6dda7920074a3074 Jul 12 17:11:48.669046 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:48.669019 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj"] Jul 12 17:11:48.671166 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:11:48.671131 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acea634_f4c9_411a_a3e6_be310b9ac0bb.slice/crio-13466a4149f3af6d4fc5e62568947bd1bd0d8b0e524a7d7a951cbceaf3fb6ecf WatchSource:0}: Error finding container 13466a4149f3af6d4fc5e62568947bd1bd0d8b0e524a7d7a951cbceaf3fb6ecf: Status 404 returned error can't find the container with id 13466a4149f3af6d4fc5e62568947bd1bd0d8b0e524a7d7a951cbceaf3fb6ecf Jul 12 17:11:49.639978 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:49.639942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" event={"ID":"7acea634-f4c9-411a-a3e6-be310b9ac0bb","Type":"ContainerStarted","Data":"13466a4149f3af6d4fc5e62568947bd1bd0d8b0e524a7d7a951cbceaf3fb6ecf"} Jul 12 17:11:49.641191 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:49.641165 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" event={"ID":"a82263e2-db57-4511-9537-7d212c8bf5ea","Type":"ContainerStarted","Data":"adffef316650407aa89140d887520a81e6db46556915bc2e6dda7920074a3074"} Jul 12 17:11:50.651253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.651091 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" event={"ID":"7acea634-f4c9-411a-a3e6-be310b9ac0bb","Type":"ContainerStarted","Data":"09efb1a882c8e99d3bfc7bd9da8a5972d0457e153e86c56b977b6d5360f69e23"} Jul 12 17:11:50.651253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.651168 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:50.653029 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.652984 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" event={"ID":"a82263e2-db57-4511-9537-7d212c8bf5ea","Type":"ContainerStarted","Data":"7945e65e1b04c720435799822107fe6df0883255d6e2fb71a1d7e22cfc36dc77"} Jul 12 17:11:50.653188 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.653047 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:11:50.654962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.654759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" event={"ID":"f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a","Type":"ContainerStarted","Data":"1e89ffebff38830eb0847b686d38d471c55dd48e164e983be178b5f18d488256"} Jul 12 17:11:50.654962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.654956 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:11:50.669836 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.669791 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" podStartSLOduration=1.9491152280000001 podStartE2EDuration="3.66977535s" totalImagesPullingTime="1.720660122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:50.668047087 +0000 UTC m=+518.162953528" watchObservedRunningTime="2026-07-12 17:11:50.66977535 +0000 UTC m=+518.164681789" Jul 12 17:11:50.684924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.684870 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" podStartSLOduration=0.875989863 podStartE2EDuration="2.684854952s" totalImagesPullingTime="1.808865089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:50.683273525 +0000 UTC m=+518.178179966" watchObservedRunningTime="2026-07-12 17:11:50.684854952 +0000 UTC m=+518.179761422" Jul 12 17:11:50.699031 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:50.698975 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" podStartSLOduration=0.950473046 podStartE2EDuration="2.698956571s" totalImagesPullingTime="1.748483525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:11:50.697666554 +0000 UTC m=+518.192573021" watchObservedRunningTime="2026-07-12 17:11:50.698956571 +0000 UTC m=+518.193863010" Jul 12 17:11:56.666453 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:56.666424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6db56bb667-z6tkj" Jul 12 17:11:58.083231 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:58.083188 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:58.181363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:58.181321 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:58.947345 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:58.947314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:11:59.047353 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:11:59.047323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:01.662132 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:01.662100 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-4qp9w" Jul 12 17:12:01.662551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:01.662270 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-k9cws" Jul 12 17:12:07.814197 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:07.814121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:15.080002 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:15.079968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:21.940801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:21.940750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:22.919085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:22.919039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:32.582862 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.582817 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2"] Jul 12 17:12:32.588863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.588843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.592401 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.592375 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 17:12:32.592546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.592405 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 12 17:12:32.592546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.592438 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5kh7x\"" Jul 12 17:12:32.592546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.592440 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 17:12:32.595659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.595624 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2"] Jul 12 17:12:32.674327 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.674295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.674509 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.674333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.674509 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.674402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97sf8\" (UniqueName: \"kubernetes.io/projected/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-kube-api-access-97sf8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.674509 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.674476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.674678 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.674512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.674678 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.674549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.684706 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.684665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:32.776006 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.775973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97sf8\" (UniqueName: \"kubernetes.io/projected/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-kube-api-access-97sf8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776493 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776582 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.776632 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.776572 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.778446 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.778427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.778604 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.778588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.783412 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.783391 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97sf8\" (UniqueName: \"kubernetes.io/projected/8585d4b6-5d3b-40a0-9d2a-4c2350e9b950-kube-api-access-97sf8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2\" (UID: \"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:32.901033 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:32.900955 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:33.045079 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:33.045046 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2"] Jul 12 17:12:33.047421 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:12:33.047379 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8585d4b6_5d3b_40a0_9d2a_4c2350e9b950.slice/crio-cb666e6806079159f2b7481e0d2b664690a6e7a95de545cc13319c3709fee046 WatchSource:0}: Error finding container cb666e6806079159f2b7481e0d2b664690a6e7a95de545cc13319c3709fee046: Status 404 returned error can't find the container with id cb666e6806079159f2b7481e0d2b664690a6e7a95de545cc13319c3709fee046 Jul 12 17:12:33.886162 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:33.886118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" event={"ID":"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950","Type":"ContainerStarted","Data":"cb666e6806079159f2b7481e0d2b664690a6e7a95de545cc13319c3709fee046"} Jul 12 17:12:34.894006 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:34.893956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" event={"ID":"8585d4b6-5d3b-40a0-9d2a-4c2350e9b950","Type":"ContainerStarted","Data":"7410389c125009373998a3c4632bfa733823e434a484f9fdafafd54d94205aab"} Jul 12 17:12:34.911245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:34.911187 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" podStartSLOduration=1.8835957269999999 podStartE2EDuration="2.911144042s" totalImagesPullingTime="1.027548315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:12:34.909986754 +0000 UTC m=+562.404893194" watchObservedRunningTime="2026-07-12 17:12:34.911144042 +0000 UTC m=+562.406050482" Jul 12 17:12:36.078267 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:36.078231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:41.406085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:41.406044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:12:42.902212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:42.902174 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:42.902623 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:42.902243 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:42.914535 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:42.914509 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:12:42.948960 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:12:42.948933 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2" Jul 12 17:13:13.396629 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:13.396600 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:13:13.399096 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:13.396600 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:13:31.946854 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:31.946818 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69c5776576-5rjdn"] Jul 12 17:13:31.950882 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:31.950859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:31.958308 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:31.957937 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69c5776576-5rjdn"] Jul 12 17:13:32.033832 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.033796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2tz\" (UniqueName: \"kubernetes.io/projected/1835fe20-3610-433c-b738-0195d1541d3e-kube-api-access-fg2tz\") pod \"authorino-69c5776576-5rjdn\" (UID: \"1835fe20-3610-433c-b738-0195d1541d3e\") " pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.034072 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.033957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1835fe20-3610-433c-b738-0195d1541d3e-tls-cert\") pod \"authorino-69c5776576-5rjdn\" (UID: \"1835fe20-3610-433c-b738-0195d1541d3e\") " pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.134867 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.134834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2tz\" (UniqueName: \"kubernetes.io/projected/1835fe20-3610-433c-b738-0195d1541d3e-kube-api-access-fg2tz\") pod \"authorino-69c5776576-5rjdn\" (UID: \"1835fe20-3610-433c-b738-0195d1541d3e\") " pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.135049 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.135029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1835fe20-3610-433c-b738-0195d1541d3e-tls-cert\") pod \"authorino-69c5776576-5rjdn\" (UID: \"1835fe20-3610-433c-b738-0195d1541d3e\") " pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.137421 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.137392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1835fe20-3610-433c-b738-0195d1541d3e-tls-cert\") pod \"authorino-69c5776576-5rjdn\" (UID: \"1835fe20-3610-433c-b738-0195d1541d3e\") " pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.142195 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.142171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2tz\" (UniqueName: \"kubernetes.io/projected/1835fe20-3610-433c-b738-0195d1541d3e-kube-api-access-fg2tz\") pod \"authorino-69c5776576-5rjdn\" (UID: \"1835fe20-3610-433c-b738-0195d1541d3e\") " pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.262097 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.262007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69c5776576-5rjdn" Jul 12 17:13:32.398826 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:32.398801 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69c5776576-5rjdn"] Jul 12 17:13:32.401315 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:13:32.401278 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1835fe20_3610_433c_b738_0195d1541d3e.slice/crio-c5c87a69badffbaddf1bee85f2c2735f74f8f06de179590e52d1143448b878d9 WatchSource:0}: Error finding container c5c87a69badffbaddf1bee85f2c2735f74f8f06de179590e52d1143448b878d9: Status 404 returned error can't find the container with id c5c87a69badffbaddf1bee85f2c2735f74f8f06de179590e52d1143448b878d9 Jul 12 17:13:33.204120 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.204073 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69c5776576-5rjdn" event={"ID":"1835fe20-3610-433c-b738-0195d1541d3e","Type":"ContainerStarted","Data":"e128c1865f10272c33b2dacaf5cc1be1e5d380d2bf633abda2db31508d0bb42f"} Jul 12 17:13:33.204638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.204127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69c5776576-5rjdn" event={"ID":"1835fe20-3610-433c-b738-0195d1541d3e","Type":"ContainerStarted","Data":"c5c87a69badffbaddf1bee85f2c2735f74f8f06de179590e52d1143448b878d9"} Jul 12 17:13:33.218607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.218542 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69c5776576-5rjdn" podStartSLOduration=1.743570962 podStartE2EDuration="2.218519929s" totalImagesPullingTime="474.948967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:13:33.218218873 +0000 UTC m=+620.713125313" watchObservedRunningTime="2026-07-12 17:13:33.218519929 +0000 UTC m=+620.713426371" Jul 12 17:13:33.259002 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.257994 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f584b7d89-7kx6f"] Jul 12 17:13:33.259002 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.258533 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f584b7d89-7kx6f" podUID="bd8ef18c-4f55-4d42-909c-5e9ee29f6506" containerName="authorino" containerID="cri-o://e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e" gracePeriod=30 Jul 12 17:13:33.530936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.530889 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:13:33.652106 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.652072 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-tls-cert\") pod \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " Jul 12 17:13:33.652262 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.652133 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpt54\" (UniqueName: \"kubernetes.io/projected/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-kube-api-access-kpt54\") pod \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\" (UID: \"bd8ef18c-4f55-4d42-909c-5e9ee29f6506\") " Jul 12 17:13:33.654334 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.654290 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-kube-api-access-kpt54" (OuterVolumeSpecName: "kube-api-access-kpt54") pod "bd8ef18c-4f55-4d42-909c-5e9ee29f6506" (UID: "bd8ef18c-4f55-4d42-909c-5e9ee29f6506"). InnerVolumeSpecName "kube-api-access-kpt54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:13:33.664762 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.664730 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bd8ef18c-4f55-4d42-909c-5e9ee29f6506" (UID: "bd8ef18c-4f55-4d42-909c-5e9ee29f6506"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:13:33.760687 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.760589 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-tls-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:13:33.760687 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.760643 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpt54\" (UniqueName: \"kubernetes.io/projected/bd8ef18c-4f55-4d42-909c-5e9ee29f6506-kube-api-access-kpt54\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:13:33.793398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:33.793344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:13:34.214185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.214138 2573 generic.go:356] "Generic (PLEG): container finished" podID="bd8ef18c-4f55-4d42-909c-5e9ee29f6506" containerID="e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e" exitCode=0 Jul 12 17:13:34.215015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.214990 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f584b7d89-7kx6f" Jul 12 17:13:34.216219 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.216185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f584b7d89-7kx6f" event={"ID":"bd8ef18c-4f55-4d42-909c-5e9ee29f6506","Type":"ContainerDied","Data":"e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e"} Jul 12 17:13:34.216346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.216237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f584b7d89-7kx6f" event={"ID":"bd8ef18c-4f55-4d42-909c-5e9ee29f6506","Type":"ContainerDied","Data":"3a17f30bc33850e68dea3b7c877c86150e8c0ee5a8ba71fdb378c494c3f90a95"} Jul 12 17:13:34.216346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.216261 2573 scope.go:117] "RemoveContainer" containerID="e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e" Jul 12 17:13:34.227311 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.227290 2573 scope.go:117] "RemoveContainer" containerID="e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e" Jul 12 17:13:34.227650 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:13:34.227629 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e\": container with ID starting with e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e not found: ID does not exist" containerID="e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e" Jul 12 17:13:34.227700 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.227665 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e"} err="failed to get container status \"e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e\": rpc error: code = NotFound desc = could not find container \"e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e\": container with ID starting with e3e05b4b7c47dcf26665aefbae3de62c1d392293f7eb26c1cdba8d73c6b6fc6e not found: ID does not exist" Jul 12 17:13:34.239454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.239422 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f584b7d89-7kx6f"] Jul 12 17:13:34.241650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:34.241623 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f584b7d89-7kx6f"] Jul 12 17:13:35.000510 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:35.000428 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:13:35.040037 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:35.040003 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8ef18c-4f55-4d42-909c-5e9ee29f6506" path="/var/lib/kubelet/pods/bd8ef18c-4f55-4d42-909c-5e9ee29f6506/volumes" Jul 12 17:13:35.098244 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:35.098191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:13:53.677433 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:53.677402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:13:58.117522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:58.117490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:13:58.218035 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:13:58.218005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:17.896273 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:17.896235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:17.996508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:17.996475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:27.391815 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:27.391781 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:27.493576 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:27.493544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:45.282897 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.282862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:45.377624 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.377585 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-h9zds"] Jul 12 17:14:45.377977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.377940 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" podUID="3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" containerName="manager" containerID="cri-o://e3ae08fb481706fa94d6354f7d2b694e4d7ee80c1b83ff44b0e1b1cfbd2201cd" gracePeriod=30 Jul 12 17:14:45.597173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.597144 2573 generic.go:356] "Generic (PLEG): container finished" podID="3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" containerID="e3ae08fb481706fa94d6354f7d2b694e4d7ee80c1b83ff44b0e1b1cfbd2201cd" exitCode=0 Jul 12 17:14:45.597352 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.597220 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" event={"ID":"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da","Type":"ContainerDied","Data":"e3ae08fb481706fa94d6354f7d2b694e4d7ee80c1b83ff44b0e1b1cfbd2201cd"} Jul 12 17:14:45.633888 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.633863 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:14:45.722017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.721975 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-webhook-cert\") pod \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " Jul 12 17:14:45.722192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.722097 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqlgg\" (UniqueName: \"kubernetes.io/projected/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-kube-api-access-pqlgg\") pod \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\" (UID: \"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da\") " Jul 12 17:14:45.723923 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.723870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" (UID: "3ee1b1d4-c9fe-46ae-adf3-f852dcf306da"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:14:45.724045 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.723997 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-kube-api-access-pqlgg" (OuterVolumeSpecName: "kube-api-access-pqlgg") pod "3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" (UID: "3ee1b1d4-c9fe-46ae-adf3-f852dcf306da"). InnerVolumeSpecName "kube-api-access-pqlgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:14:45.823552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.823470 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlgg\" (UniqueName: \"kubernetes.io/projected/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-kube-api-access-pqlgg\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:14:45.823552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:45.823501 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da-webhook-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:14:46.603490 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:46.603454 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" event={"ID":"3ee1b1d4-c9fe-46ae-adf3-f852dcf306da","Type":"ContainerDied","Data":"2c0a9b6f169adb124d8b1626253723f593bb002ef95f4997f7857362ba3e94f1"} Jul 12 17:14:46.603967 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:46.603515 2573 scope.go:117] "RemoveContainer" containerID="e3ae08fb481706fa94d6354f7d2b694e4d7ee80c1b83ff44b0e1b1cfbd2201cd" Jul 12 17:14:46.603967 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:46.603463 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b576ccbb6-h9zds" Jul 12 17:14:46.625177 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:46.625144 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-h9zds"] Jul 12 17:14:46.630954 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:46.630929 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-h9zds"] Jul 12 17:14:47.040135 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.040097 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" path="/var/lib/kubelet/pods/3ee1b1d4-c9fe-46ae-adf3-f852dcf306da/volumes" Jul 12 17:14:47.273789 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.273750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:14:47.386457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.386378 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-csl7p"] Jul 12 17:14:47.386874 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.386857 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" containerName="manager" Jul 12 17:14:47.387004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.386876 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" containerName="manager" Jul 12 17:14:47.387004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.386886 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8ef18c-4f55-4d42-909c-5e9ee29f6506" containerName="authorino" Jul 12 17:14:47.387004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.386892 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8ef18c-4f55-4d42-909c-5e9ee29f6506" containerName="authorino" Jul 12 17:14:47.387004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.387003 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8ef18c-4f55-4d42-909c-5e9ee29f6506" containerName="authorino" Jul 12 17:14:47.387166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.387013 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee1b1d4-c9fe-46ae-adf3-f852dcf306da" containerName="manager" Jul 12 17:14:47.391568 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.391545 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.395542 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.395523 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 12 17:14:47.395639 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.395561 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 17:14:47.395693 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.395641 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hx9r5\"" Jul 12 17:14:47.399427 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.399405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-csl7p"] Jul 12 17:14:47.437750 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.437723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpnc\" (UniqueName: \"kubernetes.io/projected/3f1cda9a-eedb-425e-9650-11c3997379dd-kube-api-access-gmpnc\") pod \"maas-controller-5b576ccbb6-csl7p\" (UID: \"3f1cda9a-eedb-425e-9650-11c3997379dd\") " pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.437877 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.437786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f1cda9a-eedb-425e-9650-11c3997379dd-webhook-cert\") pod \"maas-controller-5b576ccbb6-csl7p\" (UID: \"3f1cda9a-eedb-425e-9650-11c3997379dd\") " pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.539136 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.539099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f1cda9a-eedb-425e-9650-11c3997379dd-webhook-cert\") pod \"maas-controller-5b576ccbb6-csl7p\" (UID: \"3f1cda9a-eedb-425e-9650-11c3997379dd\") " pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.539338 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.539230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpnc\" (UniqueName: \"kubernetes.io/projected/3f1cda9a-eedb-425e-9650-11c3997379dd-kube-api-access-gmpnc\") pod \"maas-controller-5b576ccbb6-csl7p\" (UID: \"3f1cda9a-eedb-425e-9650-11c3997379dd\") " pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.541781 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.541747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f1cda9a-eedb-425e-9650-11c3997379dd-webhook-cert\") pod \"maas-controller-5b576ccbb6-csl7p\" (UID: \"3f1cda9a-eedb-425e-9650-11c3997379dd\") " pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.547769 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.547745 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpnc\" (UniqueName: \"kubernetes.io/projected/3f1cda9a-eedb-425e-9650-11c3997379dd-kube-api-access-gmpnc\") pod \"maas-controller-5b576ccbb6-csl7p\" (UID: \"3f1cda9a-eedb-425e-9650-11c3997379dd\") " pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.703781 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.703750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:47.840877 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.840847 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b576ccbb6-csl7p"] Jul 12 17:14:47.842734 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:14:47.842703 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1cda9a_eedb_425e_9650_11c3997379dd.slice/crio-004a0c55c9bbc380327c403cb27d19710086dec781c52a9f363ff07db86d1a55 WatchSource:0}: Error finding container 004a0c55c9bbc380327c403cb27d19710086dec781c52a9f363ff07db86d1a55: Status 404 returned error can't find the container with id 004a0c55c9bbc380327c403cb27d19710086dec781c52a9f363ff07db86d1a55 Jul 12 17:14:47.844231 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:47.844209 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 17:14:48.618082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:48.617983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" event={"ID":"3f1cda9a-eedb-425e-9650-11c3997379dd","Type":"ContainerStarted","Data":"32c7c1a6220c91366f893985c189b4c061b864dee8603e44c540bd67642b66e2"} Jul 12 17:14:48.618082 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:48.618037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" event={"ID":"3f1cda9a-eedb-425e-9650-11c3997379dd","Type":"ContainerStarted","Data":"004a0c55c9bbc380327c403cb27d19710086dec781c52a9f363ff07db86d1a55"} Jul 12 17:14:48.618266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:48.618087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:14:48.635665 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:48.635612 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" podStartSLOduration=1.22681213 podStartE2EDuration="1.635593669s" totalImagesPullingTime="408.781539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:14:48.635302345 +0000 UTC m=+696.130208785" watchObservedRunningTime="2026-07-12 17:14:48.635593669 +0000 UTC m=+696.130500109" Jul 12 17:14:59.625707 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:14:59.625676 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b576ccbb6-csl7p" Jul 12 17:15:30.196702 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:30.196664 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:30.297488 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:30.297451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:30.440985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:30.440955 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:36.213124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:36.213088 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:36.319278 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:36.319247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:36.413208 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:36.413171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:36.577314 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:36.577234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:42.282328 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:42.282298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:15:42.340283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:15:42.340247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:16:24.474889 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:16:24.474848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:18:13.428992 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:18:13.428964 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:18:13.431425 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:18:13.429002 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:19:34.659324 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:34.659222 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc"] Jul 12 17:19:34.659877 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:34.659488 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" podUID="3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" containerName="manager" containerID="cri-o://4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972" gracePeriod=10 Jul 12 17:19:37.717708 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.717683 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:19:37.775400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.775367 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-extensions-socket-volume\") pod \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " Jul 12 17:19:37.775572 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.775557 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptpwv\" (UniqueName: \"kubernetes.io/projected/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-kube-api-access-ptpwv\") pod \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\" (UID: \"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71\") " Jul 12 17:19:37.775816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.775789 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" (UID: "3c9b3a6d-0a70-4445-b43f-48fab5b5bd71"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:19:37.777537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.777510 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-kube-api-access-ptpwv" (OuterVolumeSpecName: "kube-api-access-ptpwv") pod "3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" (UID: "3c9b3a6d-0a70-4445-b43f-48fab5b5bd71"). InnerVolumeSpecName "kube-api-access-ptpwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:19:37.876358 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.876274 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptpwv\" (UniqueName: \"kubernetes.io/projected/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-kube-api-access-ptpwv\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:19:37.876358 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:37.876301 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71-extensions-socket-volume\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:19:38.175035 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.174996 2573 generic.go:356] "Generic (PLEG): container finished" podID="3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" containerID="4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972" exitCode=0 Jul 12 17:19:38.175222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.175044 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" Jul 12 17:19:38.175222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.175089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" event={"ID":"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71","Type":"ContainerDied","Data":"4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972"} Jul 12 17:19:38.175222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.175140 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc" event={"ID":"3c9b3a6d-0a70-4445-b43f-48fab5b5bd71","Type":"ContainerDied","Data":"838f71621423b2b788a64389a1853479823aaace33802288ab77bfc35158f6d5"} Jul 12 17:19:38.175222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.175162 2573 scope.go:117] "RemoveContainer" containerID="4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972" Jul 12 17:19:38.185468 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.185452 2573 scope.go:117] "RemoveContainer" containerID="4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972" Jul 12 17:19:38.185727 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:19:38.185707 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972\": container with ID starting with 4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972 not found: ID does not exist" containerID="4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972" Jul 12 17:19:38.185797 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.185735 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972"} err="failed to get container status \"4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972\": rpc error: code = NotFound desc = could not find container \"4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972\": container with ID starting with 4c8a409ff7aec258e9381ba1cc3f6e76ad7e692fd6e96b8ae2187ff7ade05972 not found: ID does not exist" Jul 12 17:19:38.196420 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.196394 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc"] Jul 12 17:19:38.202224 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:38.202190 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q75bc"] Jul 12 17:19:39.039782 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:19:39.039744 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" path="/var/lib/kubelet/pods/3c9b3a6d-0a70-4445-b43f-48fab5b5bd71/volumes" Jul 12 17:20:31.379094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:31.379051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:20:35.811751 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.811710 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn"] Jul 12 17:20:35.812187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.812173 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" containerName="manager" Jul 12 17:20:35.812232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.812190 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" containerName="manager" Jul 12 17:20:35.812285 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.812275 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9b3a6d-0a70-4445-b43f-48fab5b5bd71" containerName="manager" Jul 12 17:20:35.815455 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.815438 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:35.817957 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.817937 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wxsvq\"" Jul 12 17:20:35.824830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.824810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c1d50c1-beb6-4f98-96b2-e882ce6094b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-97kpn\" (UID: \"2c1d50c1-beb6-4f98-96b2-e882ce6094b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:35.824939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.824866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cv28\" (UniqueName: \"kubernetes.io/projected/2c1d50c1-beb6-4f98-96b2-e882ce6094b6-kube-api-access-7cv28\") pod \"kuadrant-operator-controller-manager-86878c769f-97kpn\" (UID: \"2c1d50c1-beb6-4f98-96b2-e882ce6094b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:35.833378 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.833357 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn"] Jul 12 17:20:35.925906 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.925873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c1d50c1-beb6-4f98-96b2-e882ce6094b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-97kpn\" (UID: \"2c1d50c1-beb6-4f98-96b2-e882ce6094b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:35.926123 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.925978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cv28\" (UniqueName: \"kubernetes.io/projected/2c1d50c1-beb6-4f98-96b2-e882ce6094b6-kube-api-access-7cv28\") pod \"kuadrant-operator-controller-manager-86878c769f-97kpn\" (UID: \"2c1d50c1-beb6-4f98-96b2-e882ce6094b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:35.926244 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.926222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c1d50c1-beb6-4f98-96b2-e882ce6094b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-97kpn\" (UID: \"2c1d50c1-beb6-4f98-96b2-e882ce6094b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:35.934104 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:35.934076 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cv28\" (UniqueName: \"kubernetes.io/projected/2c1d50c1-beb6-4f98-96b2-e882ce6094b6-kube-api-access-7cv28\") pod \"kuadrant-operator-controller-manager-86878c769f-97kpn\" (UID: \"2c1d50c1-beb6-4f98-96b2-e882ce6094b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:36.125449 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:36.125345 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:36.264590 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:36.264565 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn"] Jul 12 17:20:36.266931 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:20:36.266892 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1d50c1_beb6_4f98_96b2_e882ce6094b6.slice/crio-4a959a67eefc63db666be5e42abafba2a618d6307c812534b38273cac35de4c3 WatchSource:0}: Error finding container 4a959a67eefc63db666be5e42abafba2a618d6307c812534b38273cac35de4c3: Status 404 returned error can't find the container with id 4a959a67eefc63db666be5e42abafba2a618d6307c812534b38273cac35de4c3 Jul 12 17:20:36.482660 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:36.482628 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" event={"ID":"2c1d50c1-beb6-4f98-96b2-e882ce6094b6","Type":"ContainerStarted","Data":"7348bb3ce455705757a09dd500a8e40bb645378e3a72c764a583d49e96a8a1d9"} Jul 12 17:20:36.482854 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:36.482680 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" event={"ID":"2c1d50c1-beb6-4f98-96b2-e882ce6094b6","Type":"ContainerStarted","Data":"4a959a67eefc63db666be5e42abafba2a618d6307c812534b38273cac35de4c3"} Jul 12 17:20:36.482854 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:36.482750 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:20:36.502889 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:36.502831 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" podStartSLOduration=1.5028122000000002 podStartE2EDuration="1.5028122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:20:36.501763896 +0000 UTC m=+1043.996670337" watchObservedRunningTime="2026-07-12 17:20:36.5028122 +0000 UTC m=+1043.997718639" Jul 12 17:20:47.490722 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:20:47.490689 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97kpn" Jul 12 17:23:02.034614 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:23:02.034516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:23:13.461320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:23:13.461192 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:23:13.466117 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:23:13.466095 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:23:17.970229 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:23:17.970194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:23:18.022952 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:23:18.022894 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:24:21.240586 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:24:21.240543 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:24:21.341948 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:24:21.341895 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:26.845549 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:26.845510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:32.441590 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:32.441545 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:40.781305 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:40.781264 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:46.037623 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:46.037588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:49.389090 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:49.389044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:49.485129 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:49.485090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:49.908724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:49.908677 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:55.435977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:55.435938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:55.498289 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:55.498257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:55.653774 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:55.653744 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:55.717042 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:55.716950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:56.859376 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:56.859340 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:56.929558 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:56.929524 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75"] Jul 12 17:26:56.932628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:56.932604 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:56.933199 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:56.933185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:56.935801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:56.935773 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2c119f77-serving-cert\"" Jul 12 17:26:56.945344 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:56.945319 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75"] Jul 12 17:26:57.004791 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.004758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:57.008024 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.008002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabacab9-0353-4035-8a53-9843a44332ce-openshift-service-ca\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.008102 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.008061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr27f\" (UniqueName: \"kubernetes.io/projected/fabacab9-0353-4035-8a53-9843a44332ce-kube-api-access-fr27f\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.008140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.008108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.108878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.108842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr27f\" (UniqueName: \"kubernetes.io/projected/fabacab9-0353-4035-8a53-9843a44332ce-kube-api-access-fr27f\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.109094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.108944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.109094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.109004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabacab9-0353-4035-8a53-9843a44332ce-openshift-service-ca\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.109187 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:26:57.109116 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-serving-cert: secret "maas-api-e2e-ait-2c119f77-serving-cert" not found Jul 12 17:26:57.109224 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:26:57.109195 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls podName:fabacab9-0353-4035-8a53-9843a44332ce nodeName:}" failed. No retries permitted until 2026-07-12 17:26:57.609175361 +0000 UTC m=+1425.104081783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls") pod "maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" (UID: "fabacab9-0353-4035-8a53-9843a44332ce") : secret "maas-api-e2e-ait-2c119f77-serving-cert" not found Jul 12 17:26:57.109638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.109596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabacab9-0353-4035-8a53-9843a44332ce-openshift-service-ca\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.117820 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.117791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr27f\" (UniqueName: \"kubernetes.io/projected/fabacab9-0353-4035-8a53-9843a44332ce-kube-api-access-fr27f\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.296273 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.296239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:57.418452 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.418390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:57.516416 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.516377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:26:57.615054 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.615017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.617372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.617348 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls\") pod \"maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.847872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.847764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:57.989342 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.989314 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75"] Jul 12 17:26:57.991435 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:26:57.991405 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabacab9_0353_4035_8a53_9843a44332ce.slice/crio-d304a2f4466bea53136129c3f5e390290674a40ed7a8be9d9f864ba877b35089 WatchSource:0}: Error finding container d304a2f4466bea53136129c3f5e390290674a40ed7a8be9d9f864ba877b35089: Status 404 returned error can't find the container with id d304a2f4466bea53136129c3f5e390290674a40ed7a8be9d9f864ba877b35089 Jul 12 17:26:57.992884 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:57.992862 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 17:26:58.526434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:58.526392 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" event={"ID":"fabacab9-0353-4035-8a53-9843a44332ce","Type":"ContainerStarted","Data":"647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2"} Jul 12 17:26:58.526637 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:58.526459 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" event={"ID":"fabacab9-0353-4035-8a53-9843a44332ce","Type":"ContainerStarted","Data":"d304a2f4466bea53136129c3f5e390290674a40ed7a8be9d9f864ba877b35089"} Jul 12 17:26:58.526637 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:58.526509 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:26:58.545017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:58.544962 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" podStartSLOduration=2.159194618 podStartE2EDuration="2.544943093s" totalImagesPullingTime="385.748475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:26:58.543109705 +0000 UTC m=+1426.038016145" watchObservedRunningTime="2026-07-12 17:26:58.544943093 +0000 UTC m=+1426.039849567" Jul 12 17:26:59.086663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:26:59.086616 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:01.900359 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:01.900324 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp"] Jul 12 17:27:01.906140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:01.906105 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:01.909492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:01.909468 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5xzcz\"" Jul 12 17:27:01.920527 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:01.920492 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp"] Jul 12 17:27:02.053712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:02.053668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fa55b9-4177-454e-acea-0edee3da3f88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2c119f77-k97wp\" (UID: \"18fa55b9-4177-454e-acea-0edee3da3f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:02.155313 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:02.155274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fa55b9-4177-454e-acea-0edee3da3f88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2c119f77-k97wp\" (UID: \"18fa55b9-4177-454e-acea-0edee3da3f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:02.156230 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:02.156206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fa55b9-4177-454e-acea-0edee3da3f88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2c119f77-k97wp\" (UID: \"18fa55b9-4177-454e-acea-0edee3da3f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:02.216743 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:02.216703 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:02.356537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:02.356503 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp"] Jul 12 17:27:02.358749 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:27:02.358715 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fa55b9_4177_454e_acea_0edee3da3f88.slice/crio-be5f13a446092bfe42796b6e738f601b789cb04bb8de6677de3db16f0893f647 WatchSource:0}: Error finding container be5f13a446092bfe42796b6e738f601b789cb04bb8de6677de3db16f0893f647: Status 404 returned error can't find the container with id be5f13a446092bfe42796b6e738f601b789cb04bb8de6677de3db16f0893f647 Jul 12 17:27:02.548863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:02.548779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" event={"ID":"18fa55b9-4177-454e-acea-0edee3da3f88","Type":"ContainerStarted","Data":"be5f13a446092bfe42796b6e738f601b789cb04bb8de6677de3db16f0893f647"} Jul 12 17:27:04.537741 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:04.537718 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:27:05.567500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:05.567405 2573 generic.go:356] "Generic (PLEG): container finished" podID="18fa55b9-4177-454e-acea-0edee3da3f88" containerID="9be0fc165d3607d2426d293e2ac998e0bcd667b28bbfb5588bca26f052046247" exitCode=0 Jul 12 17:27:05.567500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:05.567444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" event={"ID":"18fa55b9-4177-454e-acea-0edee3da3f88","Type":"ContainerDied","Data":"9be0fc165d3607d2426d293e2ac998e0bcd667b28bbfb5588bca26f052046247"} Jul 12 17:27:06.712051 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:06.712025 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:06.902866 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:06.902801 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fa55b9-4177-454e-acea-0edee3da3f88-maas-api-service-ca\") pod \"18fa55b9-4177-454e-acea-0edee3da3f88\" (UID: \"18fa55b9-4177-454e-acea-0edee3da3f88\") " Jul 12 17:27:06.903165 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:06.903142 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fa55b9-4177-454e-acea-0edee3da3f88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18fa55b9-4177-454e-acea-0edee3da3f88" (UID: "18fa55b9-4177-454e-acea-0edee3da3f88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:27:07.003988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:07.003955 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fa55b9-4177-454e-acea-0edee3da3f88-maas-api-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:27:07.579025 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:07.578997 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" Jul 12 17:27:07.579217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:07.579039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c119f77-k97wp" event={"ID":"18fa55b9-4177-454e-acea-0edee3da3f88","Type":"ContainerDied","Data":"be5f13a446092bfe42796b6e738f601b789cb04bb8de6677de3db16f0893f647"} Jul 12 17:27:07.579217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:07.579091 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be5f13a446092bfe42796b6e738f601b789cb04bb8de6677de3db16f0893f647" Jul 12 17:27:12.276527 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.276452 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75"] Jul 12 17:27:12.276972 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.276762 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" podUID="fabacab9-0353-4035-8a53-9843a44332ce" containerName="maas-api" containerID="cri-o://647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2" gracePeriod=30 Jul 12 17:27:12.299724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.299694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:12.531362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.531299 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:27:12.552796 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.552767 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabacab9-0353-4035-8a53-9843a44332ce-openshift-service-ca\") pod \"fabacab9-0353-4035-8a53-9843a44332ce\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " Jul 12 17:27:12.552971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.552802 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr27f\" (UniqueName: \"kubernetes.io/projected/fabacab9-0353-4035-8a53-9843a44332ce-kube-api-access-fr27f\") pod \"fabacab9-0353-4035-8a53-9843a44332ce\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " Jul 12 17:27:12.552971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.552841 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls\") pod \"fabacab9-0353-4035-8a53-9843a44332ce\" (UID: \"fabacab9-0353-4035-8a53-9843a44332ce\") " Jul 12 17:27:12.553162 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.553137 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabacab9-0353-4035-8a53-9843a44332ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fabacab9-0353-4035-8a53-9843a44332ce" (UID: "fabacab9-0353-4035-8a53-9843a44332ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:27:12.553246 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.553231 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabacab9-0353-4035-8a53-9843a44332ce-openshift-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:27:12.555698 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.555672 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fabacab9-0353-4035-8a53-9843a44332ce" (UID: "fabacab9-0353-4035-8a53-9843a44332ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:27:12.555698 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.555688 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabacab9-0353-4035-8a53-9843a44332ce-kube-api-access-fr27f" (OuterVolumeSpecName: "kube-api-access-fr27f") pod "fabacab9-0353-4035-8a53-9843a44332ce" (UID: "fabacab9-0353-4035-8a53-9843a44332ce"). InnerVolumeSpecName "kube-api-access-fr27f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:27:12.604685 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.604653 2573 generic.go:356] "Generic (PLEG): container finished" podID="fabacab9-0353-4035-8a53-9843a44332ce" containerID="647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2" exitCode=0 Jul 12 17:27:12.604830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.604702 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" Jul 12 17:27:12.604830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.604743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" event={"ID":"fabacab9-0353-4035-8a53-9843a44332ce","Type":"ContainerDied","Data":"647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2"} Jul 12 17:27:12.604830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.604795 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75" event={"ID":"fabacab9-0353-4035-8a53-9843a44332ce","Type":"ContainerDied","Data":"d304a2f4466bea53136129c3f5e390290674a40ed7a8be9d9f864ba877b35089"} Jul 12 17:27:12.604830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.604815 2573 scope.go:117] "RemoveContainer" containerID="647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2" Jul 12 17:27:12.613640 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.613621 2573 scope.go:117] "RemoveContainer" containerID="647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2" Jul 12 17:27:12.613870 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:27:12.613853 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2\": container with ID starting with 647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2 not found: ID does not exist" containerID="647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2" Jul 12 17:27:12.613939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.613877 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2"} err="failed to get container status \"647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2\": rpc error: code = NotFound desc = could not find container \"647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2\": container with ID starting with 647a1366cc9d24b5a2a08620da151a01c40909d59d03a1b089dc198fd2b5fca2 not found: ID does not exist" Jul 12 17:27:12.626076 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.626057 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75"] Jul 12 17:27:12.628958 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.628939 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c119f77-8cfbff5bd-64v75"] Jul 12 17:27:12.654225 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.654202 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr27f\" (UniqueName: \"kubernetes.io/projected/fabacab9-0353-4035-8a53-9843a44332ce-kube-api-access-fr27f\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:27:12.654225 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:12.654223 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabacab9-0353-4035-8a53-9843a44332ce-maas-api-tls\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:27:13.042145 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:13.042108 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fabacab9-0353-4035-8a53-9843a44332ce" path="/var/lib/kubelet/pods/fabacab9-0353-4035-8a53-9843a44332ce/volumes" Jul 12 17:27:16.976279 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:16.976226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:22.695093 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:22.695062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:22.756375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:22.756340 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:27.161431 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:27.161393 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:27.545976 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:27.545859 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:27.646163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:27.646130 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:27.733799 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:27.733767 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:28.986886 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:28.986851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:29.061802 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.061769 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp"] Jul 12 17:27:29.062222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.062208 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabacab9-0353-4035-8a53-9843a44332ce" containerName="maas-api" Jul 12 17:27:29.062273 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.062223 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabacab9-0353-4035-8a53-9843a44332ce" containerName="maas-api" Jul 12 17:27:29.062273 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.062237 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fa55b9-4177-454e-acea-0edee3da3f88" containerName="revoke-keys" Jul 12 17:27:29.062273 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.062242 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fa55b9-4177-454e-acea-0edee3da3f88" containerName="revoke-keys" Jul 12 17:27:29.062362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.062309 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fabacab9-0353-4035-8a53-9843a44332ce" containerName="maas-api" Jul 12 17:27:29.062362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.062317 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fa55b9-4177-454e-acea-0edee3da3f88" containerName="revoke-keys" Jul 12 17:27:29.066724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.066694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.069160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.069140 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dc65b9aa-serving-cert\"" Jul 12 17:27:29.074276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.074253 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp"] Jul 12 17:27:29.098070 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.098044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:29.206347 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.206313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-maas-api-tls\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.206566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.206386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-openshift-service-ca\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.206566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.206436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825lg\" (UniqueName: \"kubernetes.io/projected/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-kube-api-access-825lg\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.298174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.298097 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:29.307341 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.307307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-maas-api-tls\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.307522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.307361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-openshift-service-ca\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.307522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.307396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825lg\" (UniqueName: \"kubernetes.io/projected/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-kube-api-access-825lg\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.308044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.308022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-openshift-service-ca\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.309707 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.309687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-maas-api-tls\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.315191 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.315173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-825lg\" (UniqueName: \"kubernetes.io/projected/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-kube-api-access-825lg\") pod \"maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.380413 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.380371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:29.526687 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.526665 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp"] Jul 12 17:27:29.528926 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:27:29.528883 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d88d8f8_80a8_43cd_9ee8_882a3ae9bd7b.slice/crio-78f7c6692e7e3bdae501621b12bb681ebc328bf3aad276a32b0f0459d88eaef7 WatchSource:0}: Error finding container 78f7c6692e7e3bdae501621b12bb681ebc328bf3aad276a32b0f0459d88eaef7: Status 404 returned error can't find the container with id 78f7c6692e7e3bdae501621b12bb681ebc328bf3aad276a32b0f0459d88eaef7 Jul 12 17:27:29.556285 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.556212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:29.657560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.657530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:29.695826 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:29.695792 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" event={"ID":"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b","Type":"ContainerStarted","Data":"78f7c6692e7e3bdae501621b12bb681ebc328bf3aad276a32b0f0459d88eaef7"} Jul 12 17:27:30.702958 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:30.702897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" event={"ID":"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b","Type":"ContainerStarted","Data":"920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693"} Jul 12 17:27:30.703346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:30.702982 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:30.720433 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:30.720379 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" podStartSLOduration=1.386209574 podStartE2EDuration="1.720363323s" totalImagesPullingTime="334.153749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:27:30.719154149 +0000 UTC m=+1458.214060588" watchObservedRunningTime="2026-07-12 17:27:30.720363323 +0000 UTC m=+1458.215269761" Jul 12 17:27:33.627943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.627884 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx"] Jul 12 17:27:33.631676 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.631656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:33.638276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.638249 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5xzcz\"" Jul 12 17:27:33.640293 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.639785 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx"] Jul 12 17:27:33.641935 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.641190 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:33.698763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.698717 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:33.751694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.751651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199be96f-2002-46c9-b3ce-b7fb056ce6c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx\" (UID: \"199be96f-2002-46c9-b3ce-b7fb056ce6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:33.853267 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.853230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199be96f-2002-46c9-b3ce-b7fb056ce6c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx\" (UID: \"199be96f-2002-46c9-b3ce-b7fb056ce6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:33.853969 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.853897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199be96f-2002-46c9-b3ce-b7fb056ce6c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx\" (UID: \"199be96f-2002-46c9-b3ce-b7fb056ce6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:33.949420 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:33.949378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:34.086383 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:34.086355 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx"] Jul 12 17:27:34.088855 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:27:34.088825 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199be96f_2002_46c9_b3ce_b7fb056ce6c3.slice/crio-d9b7e2ef4b144d32e08b07c4233b335d5953a46669ebf8fd9461f6ff2b0936c8 WatchSource:0}: Error finding container d9b7e2ef4b144d32e08b07c4233b335d5953a46669ebf8fd9461f6ff2b0936c8: Status 404 returned error can't find the container with id d9b7e2ef4b144d32e08b07c4233b335d5953a46669ebf8fd9461f6ff2b0936c8 Jul 12 17:27:34.724929 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:34.724891 2573 generic.go:356] "Generic (PLEG): container finished" podID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerID="073d6946b4b3ba9fd971774fbeda0ed5f7ad91c9bf3a3b94ff9d3078c913904f" exitCode=7 Jul 12 17:27:34.725350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:34.724987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" event={"ID":"199be96f-2002-46c9-b3ce-b7fb056ce6c3","Type":"ContainerDied","Data":"073d6946b4b3ba9fd971774fbeda0ed5f7ad91c9bf3a3b94ff9d3078c913904f"} Jul 12 17:27:34.725350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:34.725056 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" event={"ID":"199be96f-2002-46c9-b3ce-b7fb056ce6c3","Type":"ContainerStarted","Data":"d9b7e2ef4b144d32e08b07c4233b335d5953a46669ebf8fd9461f6ff2b0936c8"} Jul 12 17:27:34.725350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:34.725182 2573 scope.go:117] "RemoveContainer" containerID="073d6946b4b3ba9fd971774fbeda0ed5f7ad91c9bf3a3b94ff9d3078c913904f" Jul 12 17:27:35.731370 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:35.731335 2573 generic.go:356] "Generic (PLEG): container finished" podID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerID="597fdd9d5767e64506c19a57692e4f930726d442d80fff6d3432c2683509d6fb" exitCode=7 Jul 12 17:27:35.731829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:35.731428 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" event={"ID":"199be96f-2002-46c9-b3ce-b7fb056ce6c3","Type":"ContainerDied","Data":"597fdd9d5767e64506c19a57692e4f930726d442d80fff6d3432c2683509d6fb"} Jul 12 17:27:35.731829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:35.731489 2573 scope.go:117] "RemoveContainer" containerID="073d6946b4b3ba9fd971774fbeda0ed5f7ad91c9bf3a3b94ff9d3078c913904f" Jul 12 17:27:35.731829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:35.731719 2573 scope.go:117] "RemoveContainer" containerID="597fdd9d5767e64506c19a57692e4f930726d442d80fff6d3432c2683509d6fb" Jul 12 17:27:35.731985 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:27:35.731961 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx_odh-ai-gateway-infra(199be96f-2002-46c9-b3ce-b7fb056ce6c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" Jul 12 17:27:36.715225 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:36.715198 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:27:36.738748 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:36.738721 2573 scope.go:117] "RemoveContainer" containerID="597fdd9d5767e64506c19a57692e4f930726d442d80fff6d3432c2683509d6fb" Jul 12 17:27:36.739230 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:27:36.739001 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx_odh-ai-gateway-infra(199be96f-2002-46c9-b3ce-b7fb056ce6c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" Jul 12 17:27:37.092088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:37.092052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:27:50.033685 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:50.033657 2573 scope.go:117] "RemoveContainer" containerID="597fdd9d5767e64506c19a57692e4f930726d442d80fff6d3432c2683509d6fb" Jul 12 17:27:50.814041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:50.814009 2573 generic.go:356] "Generic (PLEG): container finished" podID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerID="3ff4de071952078b68bc9e6b0cc45600141f32368bfabadf745b7889bd4b153c" exitCode=0 Jul 12 17:27:50.814211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:50.814046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" event={"ID":"199be96f-2002-46c9-b3ce-b7fb056ce6c3","Type":"ContainerDied","Data":"3ff4de071952078b68bc9e6b0cc45600141f32368bfabadf745b7889bd4b153c"} Jul 12 17:27:50.814211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:50.814095 2573 scope.go:117] "RemoveContainer" containerID="597fdd9d5767e64506c19a57692e4f930726d442d80fff6d3432c2683509d6fb" Jul 12 17:27:51.845894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:51.845859 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx"] Jul 12 17:27:51.988202 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:51.988175 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:52.126821 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.126734 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199be96f-2002-46c9-b3ce-b7fb056ce6c3-maas-api-service-ca\") pod \"199be96f-2002-46c9-b3ce-b7fb056ce6c3\" (UID: \"199be96f-2002-46c9-b3ce-b7fb056ce6c3\") " Jul 12 17:27:52.127147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.127125 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199be96f-2002-46c9-b3ce-b7fb056ce6c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "199be96f-2002-46c9-b3ce-b7fb056ce6c3" (UID: "199be96f-2002-46c9-b3ce-b7fb056ce6c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:27:52.228320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.228285 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199be96f-2002-46c9-b3ce-b7fb056ce6c3-maas-api-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:27:52.830403 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.830365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" event={"ID":"199be96f-2002-46c9-b3ce-b7fb056ce6c3","Type":"ContainerDied","Data":"d9b7e2ef4b144d32e08b07c4233b335d5953a46669ebf8fd9461f6ff2b0936c8"} Jul 12 17:27:52.830403 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.830386 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx" Jul 12 17:27:52.830612 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.830450 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b7e2ef4b144d32e08b07c4233b335d5953a46669ebf8fd9461f6ff2b0936c8" Jul 12 17:27:52.861089 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.861055 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx"] Jul 12 17:27:52.865924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:52.865892 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-jwdjx"] Jul 12 17:27:53.040457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:27:53.040427 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" path="/var/lib/kubelet/pods/199be96f-2002-46c9-b3ce-b7fb056ce6c3/volumes" Jul 12 17:28:13.493393 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:13.493277 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:28:13.498101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:13.498083 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:28:23.688299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688253 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn"] Jul 12 17:28:23.688780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688676 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.688780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688686 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.688780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688699 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.688780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688705 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.688780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688712 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.688780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688718 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.689112 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688799 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.689112 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688808 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.689112 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.688814 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="199be96f-2002-46c9-b3ce-b7fb056ce6c3" containerName="revoke-keys" Jul 12 17:28:23.691963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.691941 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:23.694373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.694352 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5xzcz\"" Jul 12 17:28:23.700335 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.699896 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn"] Jul 12 17:28:23.823107 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.823070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7253a4-9dfb-49cb-a434-c8e53fec6890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn\" (UID: \"fa7253a4-9dfb-49cb-a434-c8e53fec6890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:23.924031 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.923997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7253a4-9dfb-49cb-a434-c8e53fec6890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn\" (UID: \"fa7253a4-9dfb-49cb-a434-c8e53fec6890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:23.924577 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:23.924558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7253a4-9dfb-49cb-a434-c8e53fec6890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn\" (UID: \"fa7253a4-9dfb-49cb-a434-c8e53fec6890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:24.005211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:24.005117 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:24.136605 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:24.136577 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn"] Jul 12 17:28:24.138903 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:28:24.138876 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7253a4_9dfb_49cb_a434_c8e53fec6890.slice/crio-fb8e939183fcf3f4425c11703904b67cce0f6494b0b95c2846ca3d720fea392b WatchSource:0}: Error finding container fb8e939183fcf3f4425c11703904b67cce0f6494b0b95c2846ca3d720fea392b: Status 404 returned error can't find the container with id fb8e939183fcf3f4425c11703904b67cce0f6494b0b95c2846ca3d720fea392b Jul 12 17:28:25.014795 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:25.014759 2573 generic.go:356] "Generic (PLEG): container finished" podID="fa7253a4-9dfb-49cb-a434-c8e53fec6890" containerID="a624ec4f366a02c20f93d2ba874a96de211bd716144723e21247bf057ff483cf" exitCode=0 Jul 12 17:28:25.015177 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:25.014836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" event={"ID":"fa7253a4-9dfb-49cb-a434-c8e53fec6890","Type":"ContainerDied","Data":"a624ec4f366a02c20f93d2ba874a96de211bd716144723e21247bf057ff483cf"} Jul 12 17:28:25.015177 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:25.014882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" event={"ID":"fa7253a4-9dfb-49cb-a434-c8e53fec6890","Type":"ContainerStarted","Data":"fb8e939183fcf3f4425c11703904b67cce0f6494b0b95c2846ca3d720fea392b"} Jul 12 17:28:26.148297 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:26.148273 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:26.245822 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:26.245790 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7253a4-9dfb-49cb-a434-c8e53fec6890-maas-api-service-ca\") pod \"fa7253a4-9dfb-49cb-a434-c8e53fec6890\" (UID: \"fa7253a4-9dfb-49cb-a434-c8e53fec6890\") " Jul 12 17:28:26.246185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:26.246163 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7253a4-9dfb-49cb-a434-c8e53fec6890-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa7253a4-9dfb-49cb-a434-c8e53fec6890" (UID: "fa7253a4-9dfb-49cb-a434-c8e53fec6890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:28:26.346851 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:26.346761 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7253a4-9dfb-49cb-a434-c8e53fec6890-maas-api-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:27.027329 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:27.027303 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" Jul 12 17:28:27.027507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:27.027303 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn" event={"ID":"fa7253a4-9dfb-49cb-a434-c8e53fec6890","Type":"ContainerDied","Data":"fb8e939183fcf3f4425c11703904b67cce0f6494b0b95c2846ca3d720fea392b"} Jul 12 17:28:27.027507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:27.027417 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb8e939183fcf3f4425c11703904b67cce0f6494b0b95c2846ca3d720fea392b" Jul 12 17:28:28.937557 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:28.937519 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:29.038570 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:29.038537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:33.786191 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:33.786153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:33.919266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:33.919224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:34.044446 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.044364 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp"] Jul 12 17:28:34.044872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.044836 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" podUID="0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" containerName="maas-api" containerID="cri-o://920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693" gracePeriod=30 Jul 12 17:28:34.067132 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.067099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:34.311218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.311193 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:28:34.321062 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.321039 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-maas-api-tls\") pod \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " Jul 12 17:28:34.321140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.321098 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-openshift-service-ca\") pod \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " Jul 12 17:28:34.321140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.321131 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-825lg\" (UniqueName: \"kubernetes.io/projected/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-kube-api-access-825lg\") pod \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\" (UID: \"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b\") " Jul 12 17:28:34.321447 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.321426 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" (UID: "0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:28:34.323091 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.323071 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-kube-api-access-825lg" (OuterVolumeSpecName: "kube-api-access-825lg") pod "0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" (UID: "0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b"). InnerVolumeSpecName "kube-api-access-825lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:28:34.323181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.323163 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" (UID: "0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:28:34.422516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.422474 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-maas-api-tls\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:34.422686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.422531 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-openshift-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:34.422686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:34.422543 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825lg\" (UniqueName: \"kubernetes.io/projected/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b-kube-api-access-825lg\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:35.070323 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.070279 2573 generic.go:356] "Generic (PLEG): container finished" podID="0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" containerID="920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693" exitCode=0 Jul 12 17:28:35.070778 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.070346 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" Jul 12 17:28:35.070778 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.070359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" event={"ID":"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b","Type":"ContainerDied","Data":"920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693"} Jul 12 17:28:35.070778 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.070413 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp" event={"ID":"0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b","Type":"ContainerDied","Data":"78f7c6692e7e3bdae501621b12bb681ebc328bf3aad276a32b0f0459d88eaef7"} Jul 12 17:28:35.070778 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.070435 2573 scope.go:117] "RemoveContainer" containerID="920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693" Jul 12 17:28:35.079450 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.079434 2573 scope.go:117] "RemoveContainer" containerID="920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693" Jul 12 17:28:35.079691 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:28:35.079673 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693\": container with ID starting with 920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693 not found: ID does not exist" containerID="920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693" Jul 12 17:28:35.079751 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.079699 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693"} err="failed to get container status \"920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693\": rpc error: code = NotFound desc = could not find container \"920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693\": container with ID starting with 920ee0e2bc53d5ddae03b5dfbb7160deaf75474bd852356ac3dac444f7b81693 not found: ID does not exist" Jul 12 17:28:35.086679 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.086658 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp"] Jul 12 17:28:35.090396 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:35.090377 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc65b9aa-6f76798b7b-klvxp"] Jul 12 17:28:37.040702 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:37.040666 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" path="/var/lib/kubelet/pods/0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b/volumes" Jul 12 17:28:39.699090 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:39.699057 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:39.799586 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:39.799547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:39.917723 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:39.917695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:40.131026 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:40.130994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:40.231962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:40.231931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:41.028272 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.028241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:41.101716 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.101684 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7"] Jul 12 17:28:41.102339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.102313 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" containerName="maas-api" Jul 12 17:28:41.102339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.102338 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" containerName="maas-api" Jul 12 17:28:41.102603 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.102374 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7253a4-9dfb-49cb-a434-c8e53fec6890" containerName="revoke-keys" Jul 12 17:28:41.102603 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.102384 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7253a4-9dfb-49cb-a434-c8e53fec6890" containerName="revoke-keys" Jul 12 17:28:41.102603 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.102514 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa7253a4-9dfb-49cb-a434-c8e53fec6890" containerName="revoke-keys" Jul 12 17:28:41.102603 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.102530 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d88d8f8-80a8-43cd-9ee8-882a3ae9bd7b" containerName="maas-api" Jul 12 17:28:41.107375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.107353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.109761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.109737 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-187a6623-serving-cert\"" Jul 12 17:28:41.117589 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.117568 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7"] Jul 12 17:28:41.120531 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.120505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:41.180221 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.180186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:41.183421 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.183401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-maas-api-tls\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.183507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.183440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6xvt\" (UniqueName: \"kubernetes.io/projected/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-kube-api-access-k6xvt\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.183507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.183461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-openshift-service-ca\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.284509 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.284423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-maas-api-tls\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.284509 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.284468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6xvt\" (UniqueName: \"kubernetes.io/projected/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-kube-api-access-k6xvt\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.284710 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.284554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-openshift-service-ca\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.285230 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.285206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-openshift-service-ca\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.286777 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.286758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-maas-api-tls\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.295080 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.295060 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6xvt\" (UniqueName: \"kubernetes.io/projected/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-kube-api-access-k6xvt\") pod \"maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.313088 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.313065 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:41.420255 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.420221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:41.561933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.561792 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7"] Jul 12 17:28:41.564242 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:28:41.564210 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1588b746_d81e_4c9a_9892_f3fcdf48ecb9.slice/crio-43bbe1ba700f7dc588c5b839ad99400e1352a5c74cb8313b21742b76474366cd WatchSource:0}: Error finding container 43bbe1ba700f7dc588c5b839ad99400e1352a5c74cb8313b21742b76474366cd: Status 404 returned error can't find the container with id 43bbe1ba700f7dc588c5b839ad99400e1352a5c74cb8313b21742b76474366cd Jul 12 17:28:41.587132 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.587101 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:41.686777 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:41.686751 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:42.112009 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:42.111973 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" event={"ID":"1588b746-d81e-4c9a-9892-f3fcdf48ecb9","Type":"ContainerStarted","Data":"41e789e693f32be1854d58915ab2084ec5fa964fb7a61ecfd864306db7a37acc"} Jul 12 17:28:42.112396 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:42.112025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" event={"ID":"1588b746-d81e-4c9a-9892-f3fcdf48ecb9","Type":"ContainerStarted","Data":"43bbe1ba700f7dc588c5b839ad99400e1352a5c74cb8313b21742b76474366cd"} Jul 12 17:28:42.112396 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:42.112121 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:42.127924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:42.127875 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" podStartSLOduration=0.698505842 podStartE2EDuration="1.127860112s" totalImagesPullingTime="429.35427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:28:42.126441503 +0000 UTC m=+1529.621347944" watchObservedRunningTime="2026-07-12 17:28:42.127860112 +0000 UTC m=+1529.622766551" Jul 12 17:28:43.427662 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:43.427615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:46.661847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.661810 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk"] Jul 12 17:28:46.666323 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.666297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:46.668776 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.668749 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5xzcz\"" Jul 12 17:28:46.670406 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.670377 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk"] Jul 12 17:28:46.735653 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.735621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51b0fd0-2710-4110-849f-21c5cd205ee3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-187a6623-jl9jk\" (UID: \"b51b0fd0-2710-4110-849f-21c5cd205ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:46.836578 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.836538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51b0fd0-2710-4110-849f-21c5cd205ee3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-187a6623-jl9jk\" (UID: \"b51b0fd0-2710-4110-849f-21c5cd205ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:46.837402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.837378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51b0fd0-2710-4110-849f-21c5cd205ee3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-187a6623-jl9jk\" (UID: \"b51b0fd0-2710-4110-849f-21c5cd205ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:46.980011 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:46.979894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:47.128714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:47.128691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk"] Jul 12 17:28:47.130194 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:28:47.130167 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51b0fd0_2710_4110_849f_21c5cd205ee3.slice/crio-9fc633b6736b4781b906c5a6c98d9d53198bf762b04505a0e4db497d6e6d53eb WatchSource:0}: Error finding container 9fc633b6736b4781b906c5a6c98d9d53198bf762b04505a0e4db497d6e6d53eb: Status 404 returned error can't find the container with id 9fc633b6736b4781b906c5a6c98d9d53198bf762b04505a0e4db497d6e6d53eb Jul 12 17:28:47.141553 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:47.141525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" event={"ID":"b51b0fd0-2710-4110-849f-21c5cd205ee3","Type":"ContainerStarted","Data":"9fc633b6736b4781b906c5a6c98d9d53198bf762b04505a0e4db497d6e6d53eb"} Jul 12 17:28:48.122833 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:48.122754 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:48.148563 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:48.148484 2573 generic.go:356] "Generic (PLEG): container finished" podID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerID="d0aa4e0c70922340cdf6347caf924d2227a2fba98ca359fce817c43819b2c270" exitCode=7 Jul 12 17:28:48.148740 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:48.148581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" event={"ID":"b51b0fd0-2710-4110-849f-21c5cd205ee3","Type":"ContainerDied","Data":"d0aa4e0c70922340cdf6347caf924d2227a2fba98ca359fce817c43819b2c270"} Jul 12 17:28:48.149144 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:48.149123 2573 scope.go:117] "RemoveContainer" containerID="d0aa4e0c70922340cdf6347caf924d2227a2fba98ca359fce817c43819b2c270" Jul 12 17:28:49.156424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:49.156386 2573 generic.go:356] "Generic (PLEG): container finished" podID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerID="35c0d2ecafc28c62116e321b0b7ef59d67f2a43349ccb5cf5973c65bbf8ffb88" exitCode=0 Jul 12 17:28:49.156850 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:49.156448 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" event={"ID":"b51b0fd0-2710-4110-849f-21c5cd205ee3","Type":"ContainerDied","Data":"35c0d2ecafc28c62116e321b0b7ef59d67f2a43349ccb5cf5973c65bbf8ffb88"} Jul 12 17:28:49.156850 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:49.156507 2573 scope.go:117] "RemoveContainer" containerID="d0aa4e0c70922340cdf6347caf924d2227a2fba98ca359fce817c43819b2c270" Jul 12 17:28:50.316372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:50.316340 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:50.369720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:50.369696 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51b0fd0-2710-4110-849f-21c5cd205ee3-maas-api-service-ca\") pod \"b51b0fd0-2710-4110-849f-21c5cd205ee3\" (UID: \"b51b0fd0-2710-4110-849f-21c5cd205ee3\") " Jul 12 17:28:50.370036 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:50.370012 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51b0fd0-2710-4110-849f-21c5cd205ee3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b51b0fd0-2710-4110-849f-21c5cd205ee3" (UID: "b51b0fd0-2710-4110-849f-21c5cd205ee3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:28:50.471342 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:50.471223 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51b0fd0-2710-4110-849f-21c5cd205ee3-maas-api-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:51.172812 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:51.172784 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" Jul 12 17:28:51.173000 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:51.172810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-187a6623-jl9jk" event={"ID":"b51b0fd0-2710-4110-849f-21c5cd205ee3","Type":"ContainerDied","Data":"9fc633b6736b4781b906c5a6c98d9d53198bf762b04505a0e4db497d6e6d53eb"} Jul 12 17:28:51.173000 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:51.172852 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc633b6736b4781b906c5a6c98d9d53198bf762b04505a0e4db497d6e6d53eb" Jul 12 17:28:51.258761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:51.258726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:56.992886 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:56.992854 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7"] Jul 12 17:28:56.993726 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:56.993692 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" podUID="1588b746-d81e-4c9a-9892-f3fcdf48ecb9" containerName="maas-api" containerID="cri-o://41e789e693f32be1854d58915ab2084ec5fa964fb7a61ecfd864306db7a37acc" gracePeriod=30 Jul 12 17:28:57.019519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.019483 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:28:57.203850 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.203821 2573 generic.go:356] "Generic (PLEG): container finished" podID="1588b746-d81e-4c9a-9892-f3fcdf48ecb9" containerID="41e789e693f32be1854d58915ab2084ec5fa964fb7a61ecfd864306db7a37acc" exitCode=0 Jul 12 17:28:57.204059 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.203894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" event={"ID":"1588b746-d81e-4c9a-9892-f3fcdf48ecb9","Type":"ContainerDied","Data":"41e789e693f32be1854d58915ab2084ec5fa964fb7a61ecfd864306db7a37acc"} Jul 12 17:28:57.250406 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.250346 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:57.335492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.335460 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-openshift-service-ca\") pod \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " Jul 12 17:28:57.335662 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.335519 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-maas-api-tls\") pod \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " Jul 12 17:28:57.335662 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.335564 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6xvt\" (UniqueName: \"kubernetes.io/projected/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-kube-api-access-k6xvt\") pod \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\" (UID: \"1588b746-d81e-4c9a-9892-f3fcdf48ecb9\") " Jul 12 17:28:57.335906 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.335880 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1588b746-d81e-4c9a-9892-f3fcdf48ecb9" (UID: "1588b746-d81e-4c9a-9892-f3fcdf48ecb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:28:57.337615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.337586 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1588b746-d81e-4c9a-9892-f3fcdf48ecb9" (UID: "1588b746-d81e-4c9a-9892-f3fcdf48ecb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:28:57.337739 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.337589 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-kube-api-access-k6xvt" (OuterVolumeSpecName: "kube-api-access-k6xvt") pod "1588b746-d81e-4c9a-9892-f3fcdf48ecb9" (UID: "1588b746-d81e-4c9a-9892-f3fcdf48ecb9"). InnerVolumeSpecName "kube-api-access-k6xvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:28:57.436988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.436947 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-maas-api-tls\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:57.436988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.436980 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6xvt\" (UniqueName: \"kubernetes.io/projected/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-kube-api-access-k6xvt\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:57.436988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:57.436989 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1588b746-d81e-4c9a-9892-f3fcdf48ecb9-openshift-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:28:58.210452 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:58.210418 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" Jul 12 17:28:58.210889 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:58.210455 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7" event={"ID":"1588b746-d81e-4c9a-9892-f3fcdf48ecb9","Type":"ContainerDied","Data":"43bbe1ba700f7dc588c5b839ad99400e1352a5c74cb8313b21742b76474366cd"} Jul 12 17:28:58.210889 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:58.210514 2573 scope.go:117] "RemoveContainer" containerID="41e789e693f32be1854d58915ab2084ec5fa964fb7a61ecfd864306db7a37acc" Jul 12 17:28:58.230252 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:58.230221 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7"] Jul 12 17:28:58.234658 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:58.234630 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-187a6623-7bcb68f77d-5zws7"] Jul 12 17:28:59.040853 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:28:59.040812 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1588b746-d81e-4c9a-9892-f3fcdf48ecb9" path="/var/lib/kubelet/pods/1588b746-d81e-4c9a-9892-f3fcdf48ecb9/volumes" Jul 12 17:29:01.730053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:01.730020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:07.269651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:07.269617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:07.342035 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:07.342003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:12.748638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:12.748603 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:13.020264 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020183 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb"] Jul 12 17:29:13.020742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020720 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerName="revoke-keys" Jul 12 17:29:13.020842 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020743 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerName="revoke-keys" Jul 12 17:29:13.020842 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020758 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1588b746-d81e-4c9a-9892-f3fcdf48ecb9" containerName="maas-api" Jul 12 17:29:13.020842 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020765 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1588b746-d81e-4c9a-9892-f3fcdf48ecb9" containerName="maas-api" Jul 12 17:29:13.021020 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020889 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1588b746-d81e-4c9a-9892-f3fcdf48ecb9" containerName="maas-api" Jul 12 17:29:13.021020 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.020900 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerName="revoke-keys" Jul 12 17:29:13.025480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.025456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.028083 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.028057 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-99b987c8-openshift-default-dockercfg-7j7kd\"" Jul 12 17:29:13.028203 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.028058 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 17:29:13.044214 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.044184 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:13.046958 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.046937 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb"] Jul 12 17:29:13.185070 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185026 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd02b732-e938-4883-8638-24994555f0ea-istio-podinfo\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185281 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-socket\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185281 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-credential-socket\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185281 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185180 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-data\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94rf8\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-kube-api-access-94rf8\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-certs\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185550 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-istio-token\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185550 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd02b732-e938-4883-8638-24994555f0ea-istiod-ca-cert\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.185617 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.185551 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-envoy\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.286761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.286660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94rf8\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-kube-api-access-94rf8\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.286761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.286709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-certs\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.286761 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.286732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-istio-token\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287051 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.286847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd02b732-e938-4883-8638-24994555f0ea-istiod-ca-cert\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287051 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.286889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-envoy\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287051 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd02b732-e938-4883-8638-24994555f0ea-istio-podinfo\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-socket\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-certs\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-credential-socket\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-data\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287420 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-socket\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287478 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287422 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-credential-socket\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287530 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-data\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.287623 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.287604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd02b732-e938-4883-8638-24994555f0ea-istiod-ca-cert\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.289226 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.289204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-envoy\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.289540 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.289519 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd02b732-e938-4883-8638-24994555f0ea-istio-podinfo\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.296893 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.296860 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94rf8\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-kube-api-access-94rf8\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.302444 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.302415 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-istio-token\") pod \"e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.346584 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.346549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:13.503865 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:13.503840 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb"] Jul 12 17:29:13.505707 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:29:13.505680 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd02b732_e938_4883_8638_24994555f0ea.slice/crio-a11146621fc7e3d4afdee6dcb6251f4166536ebe0fb3f3e7e8a7d7768aa48bfc WatchSource:0}: Error finding container a11146621fc7e3d4afdee6dcb6251f4166536ebe0fb3f3e7e8a7d7768aa48bfc: Status 404 returned error can't find the container with id a11146621fc7e3d4afdee6dcb6251f4166536ebe0fb3f3e7e8a7d7768aa48bfc Jul 12 17:29:14.192049 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.192018 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:14.251020 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.250989 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5"] Jul 12 17:29:14.251648 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.251622 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerName="revoke-keys" Jul 12 17:29:14.251648 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.251649 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerName="revoke-keys" Jul 12 17:29:14.251816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.251755 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51b0fd0-2710-4110-849f-21c5cd205ee3" containerName="revoke-keys" Jul 12 17:29:14.255086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.255067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.257667 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.257644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-99b987c8-serving-cert\"" Jul 12 17:29:14.266670 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.266648 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5"] Jul 12 17:29:14.268218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.268194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:14.299167 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.299141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" event={"ID":"cd02b732-e938-4883-8638-24994555f0ea","Type":"ContainerStarted","Data":"a11146621fc7e3d4afdee6dcb6251f4166536ebe0fb3f3e7e8a7d7768aa48bfc"} Jul 12 17:29:14.332816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.332789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:14.398296 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.398253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxsr\" (UniqueName: \"kubernetes.io/projected/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-kube-api-access-9mxsr\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.398485 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.398453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.398575 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.398555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-openshift-service-ca\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.499469 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.499376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.499469 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.499460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-openshift-service-ca\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.499699 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:29:14.499494 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-serving-cert: secret "maas-api-e2e-derive-99b987c8-serving-cert" not found Jul 12 17:29:14.499699 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:29:14.499579 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls podName:5d831443-7f9e-4718-8d60-3b7fbfd2eefa nodeName:}" failed. No retries permitted until 2026-07-12 17:29:14.999557554 +0000 UTC m=+1562.494463984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls") pod "maas-api-e2e-derive-99b987c8-549c987486-kxmz5" (UID: "5d831443-7f9e-4718-8d60-3b7fbfd2eefa") : secret "maas-api-e2e-derive-99b987c8-serving-cert" not found Jul 12 17:29:14.499699 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.499496 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxsr\" (UniqueName: \"kubernetes.io/projected/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-kube-api-access-9mxsr\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.500212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.500195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-openshift-service-ca\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.516547 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.516521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxsr\" (UniqueName: \"kubernetes.io/projected/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-kube-api-access-9mxsr\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:14.680845 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.680812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:14.839719 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.839631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:14.937277 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:14.937242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:15.005418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:15.005385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:15.008410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:15.008381 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls\") pod \"maas-api-e2e-derive-99b987c8-549c987486-kxmz5\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:15.167310 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:15.167269 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:15.321791 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:15.321760 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5"] Jul 12 17:29:16.217206 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:29:16.217164 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d831443_7f9e_4718_8d60_3b7fbfd2eefa.slice/crio-4c3265e25dcb35b8384d5843c77cdb0d937980c5fcbc3f6ed0743fc4485a467d WatchSource:0}: Error finding container 4c3265e25dcb35b8384d5843c77cdb0d937980c5fcbc3f6ed0743fc4485a467d: Status 404 returned error can't find the container with id 4c3265e25dcb35b8384d5843c77cdb0d937980c5fcbc3f6ed0743fc4485a467d Jul 12 17:29:16.294286 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:16.294254 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:29:16.294373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:16.294320 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:29:16.294373 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:16.294350 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:29:16.313186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:16.313111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" event={"ID":"5d831443-7f9e-4718-8d60-3b7fbfd2eefa","Type":"ContainerStarted","Data":"4c3265e25dcb35b8384d5843c77cdb0d937980c5fcbc3f6ed0743fc4485a467d"} Jul 12 17:29:17.320473 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.320441 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" event={"ID":"cd02b732-e938-4883-8638-24994555f0ea","Type":"ContainerStarted","Data":"06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f"} Jul 12 17:29:17.322049 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.322020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" event={"ID":"5d831443-7f9e-4718-8d60-3b7fbfd2eefa","Type":"ContainerStarted","Data":"690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b"} Jul 12 17:29:17.322189 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.322072 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:17.341712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.341657 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" podStartSLOduration=1.5553826640000001 podStartE2EDuration="4.341639671s" totalImagesPullingTime="2.786257007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:29:17.338216239 +0000 UTC m=+1564.833122683" watchObservedRunningTime="2026-07-12 17:29:17.341639671 +0000 UTC m=+1564.836546110" Jul 12 17:29:17.346733 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.346697 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:17.351683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.351655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:17.356264 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:17.356214 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" podStartSLOduration=2.986497106 podStartE2EDuration="3.356198461s" totalImagesPullingTime="369.701355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:29:17.353952493 +0000 UTC m=+1564.848858933" watchObservedRunningTime="2026-07-12 17:29:17.356198461 +0000 UTC m=+1564.851104927" Jul 12 17:29:18.327193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:18.327093 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:18.339935 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:18.335198 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:19.112745 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:19.112712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:19.166699 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:19.166659 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:22.497325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:22.497293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:29:23.332988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:23.332957 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:29:25.377119 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:25.377019 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd02b732-e938-4883-8638-24994555f0ea" containerID="06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f" exitCode=0 Jul 12 17:29:25.377119 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:25.377084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" event={"ID":"cd02b732-e938-4883-8638-24994555f0ea","Type":"ContainerDied","Data":"06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f"} Jul 12 17:29:25.377527 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:25.377475 2573 scope.go:117] "RemoveContainer" containerID="06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f" Jul 12 17:29:25.379442 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:25.379412 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:29:25.379552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:25.379494 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:29:25.379552 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:25.379543 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:29:26.384664 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:26.384627 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" event={"ID":"cd02b732-e938-4883-8638-24994555f0ea","Type":"ContainerStarted","Data":"933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea"} Jul 12 17:29:27.347587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:27.347548 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:27.347972 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:27.347949 2573 patch_prober.go:28] interesting pod/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 17:29:27.348192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:27.348167 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 17:29:28.347625 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:28.347583 2573 patch_prober.go:28] interesting pod/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 17:29:28.348161 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:28.347654 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 17:29:29.347724 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:29.347684 2573 patch_prober.go:28] interesting pod/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 17:29:29.348363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:29.348329 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 17:29:30.351364 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:30.351332 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:30.351806 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:30.351594 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:29:30.352343 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:29:30.352325 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:30:11.155817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:11.155783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:11.257235 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:11.257196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:19.089867 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:19.089829 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:19.186653 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:19.186617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:28.969144 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:28.969108 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5"] Jul 12 17:30:28.970086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:28.969894 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" podUID="5d831443-7f9e-4718-8d60-3b7fbfd2eefa" containerName="maas-api" containerID="cri-o://690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b" gracePeriod=30 Jul 12 17:30:28.995730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:28.995701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:29.223037 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.222975 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:30:29.338527 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.338496 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-openshift-service-ca\") pod \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " Jul 12 17:30:29.338710 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.338567 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mxsr\" (UniqueName: \"kubernetes.io/projected/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-kube-api-access-9mxsr\") pod \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " Jul 12 17:30:29.338710 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.338591 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls\") pod \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\" (UID: \"5d831443-7f9e-4718-8d60-3b7fbfd2eefa\") " Jul 12 17:30:29.339122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.339095 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d831443-7f9e-4718-8d60-3b7fbfd2eefa" (UID: "5d831443-7f9e-4718-8d60-3b7fbfd2eefa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:30:29.341211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.341184 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d831443-7f9e-4718-8d60-3b7fbfd2eefa" (UID: "5d831443-7f9e-4718-8d60-3b7fbfd2eefa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:30:29.341320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.341302 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-kube-api-access-9mxsr" (OuterVolumeSpecName: "kube-api-access-9mxsr") pod "5d831443-7f9e-4718-8d60-3b7fbfd2eefa" (UID: "5d831443-7f9e-4718-8d60-3b7fbfd2eefa"). InnerVolumeSpecName "kube-api-access-9mxsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:30:29.440163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.440126 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-openshift-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:29.440163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.440162 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxsr\" (UniqueName: \"kubernetes.io/projected/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-kube-api-access-9mxsr\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:29.440350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.440176 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d831443-7f9e-4718-8d60-3b7fbfd2eefa-maas-api-tls\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:29.726013 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.725981 2573 generic.go:356] "Generic (PLEG): container finished" podID="5d831443-7f9e-4718-8d60-3b7fbfd2eefa" containerID="690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b" exitCode=0 Jul 12 17:30:29.726213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.726032 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" Jul 12 17:30:29.726213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.726065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" event={"ID":"5d831443-7f9e-4718-8d60-3b7fbfd2eefa","Type":"ContainerDied","Data":"690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b"} Jul 12 17:30:29.726213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.726127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5" event={"ID":"5d831443-7f9e-4718-8d60-3b7fbfd2eefa","Type":"ContainerDied","Data":"4c3265e25dcb35b8384d5843c77cdb0d937980c5fcbc3f6ed0743fc4485a467d"} Jul 12 17:30:29.726213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.726150 2573 scope.go:117] "RemoveContainer" containerID="690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b" Jul 12 17:30:29.740053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.740038 2573 scope.go:117] "RemoveContainer" containerID="690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b" Jul 12 17:30:29.740329 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:30:29.740308 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b\": container with ID starting with 690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b not found: ID does not exist" containerID="690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b" Jul 12 17:30:29.740405 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.740340 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b"} err="failed to get container status \"690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b\": rpc error: code = NotFound desc = could not find container \"690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b\": container with ID starting with 690b10f6ac318dc626100956c2bb985338f8710f3cb711ad05a836d733f54e2b not found: ID does not exist" Jul 12 17:30:29.753859 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.753838 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5"] Jul 12 17:30:29.757733 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:29.757711 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99b987c8-549c987486-kxmz5"] Jul 12 17:30:31.040693 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:31.040657 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d831443-7f9e-4718-8d60-3b7fbfd2eefa" path="/var/lib/kubelet/pods/5d831443-7f9e-4718-8d60-3b7fbfd2eefa/volumes" Jul 12 17:30:33.718657 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:33.718620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:39.445682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:39.445640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:39.508129 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:39.508096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:43.875668 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:43.875611 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb"] Jul 12 17:30:43.876243 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:43.875968 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" containerID="cri-o://933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea" gracePeriod=30 Jul 12 17:30:43.902841 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:43.902807 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:44.535358 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:44.535323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:44.674519 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:44.674479 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:44.768200 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:44.768166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:45.354286 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:45.354251 2573 patch_prober.go:28] interesting pod/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 17:30:45.354674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:45.354303 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 17:30:46.227327 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:46.227287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:49.123276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.123245 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:30:49.215735 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215699 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd02b732-e938-4883-8638-24994555f0ea-istiod-ca-cert\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.215871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215781 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-data\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.215871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215809 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-socket\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.215871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215834 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-certs\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.216036 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215883 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-istio-token\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.216036 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215950 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd02b732-e938-4883-8638-24994555f0ea-istio-podinfo\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.216036 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.215977 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94rf8\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-kube-api-access-94rf8\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.216036 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216030 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-credential-socket\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.216231 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216056 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-envoy\") pod \"cd02b732-e938-4883-8638-24994555f0ea\" (UID: \"cd02b732-e938-4883-8638-24994555f0ea\") " Jul 12 17:30:49.216231 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216163 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:30:49.216231 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216203 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:30:49.216387 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216221 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd02b732-e938-4883-8638-24994555f0ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:30:49.216507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216435 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:30:49.216507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216499 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd02b732-e938-4883-8638-24994555f0ea-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.216628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216519 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.216628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216532 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.216805 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.216781 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:30:49.218203 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.218176 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-kube-api-access-94rf8" (OuterVolumeSpecName: "kube-api-access-94rf8") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "kube-api-access-94rf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:30:49.218346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.218203 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:30:49.218346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.218276 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:30:49.218572 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.218549 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd02b732-e938-4883-8638-24994555f0ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd02b732-e938-4883-8638-24994555f0ea" (UID: "cd02b732-e938-4883-8638-24994555f0ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:30:49.317922 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.317872 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.318108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.317941 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.318108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.317951 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd02b732-e938-4883-8638-24994555f0ea-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.318108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.317960 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94rf8\" (UniqueName: \"kubernetes.io/projected/cd02b732-e938-4883-8638-24994555f0ea-kube-api-access-94rf8\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.318108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.317970 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.318108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.317979 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd02b732-e938-4883-8638-24994555f0ea-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:30:49.836127 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.836092 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd02b732-e938-4883-8638-24994555f0ea" containerID="933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea" exitCode=0 Jul 12 17:30:49.836320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.836149 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" Jul 12 17:30:49.836320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.836180 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" event={"ID":"cd02b732-e938-4883-8638-24994555f0ea","Type":"ContainerDied","Data":"933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea"} Jul 12 17:30:49.836320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.836281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb" event={"ID":"cd02b732-e938-4883-8638-24994555f0ea","Type":"ContainerDied","Data":"a11146621fc7e3d4afdee6dcb6251f4166536ebe0fb3f3e7e8a7d7768aa48bfc"} Jul 12 17:30:49.836320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.836314 2573 scope.go:117] "RemoveContainer" containerID="933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea" Jul 12 17:30:49.846053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.846032 2573 scope.go:117] "RemoveContainer" containerID="06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f" Jul 12 17:30:49.854586 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.854568 2573 scope.go:117] "RemoveContainer" containerID="933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea" Jul 12 17:30:49.854884 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:30:49.854858 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea\": container with ID starting with 933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea not found: ID does not exist" containerID="933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea" Jul 12 17:30:49.854994 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.854898 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea"} err="failed to get container status \"933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea\": rpc error: code = NotFound desc = could not find container \"933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea\": container with ID starting with 933c2c05e3bb3033cae2532ccf46ae50cbc08b187fc3794e537d0f5455e0e0ea not found: ID does not exist" Jul 12 17:30:49.854994 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.854947 2573 scope.go:117] "RemoveContainer" containerID="06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f" Jul 12 17:30:49.855248 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:30:49.855221 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f\": container with ID starting with 06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f not found: ID does not exist" containerID="06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f" Jul 12 17:30:49.855325 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.855255 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f"} err="failed to get container status \"06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f\": rpc error: code = NotFound desc = could not find container \"06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f\": container with ID starting with 06e2662644cf1b99855ba8dcee971c54f4f559dbb412b8fbda4221ea142e148f not found: ID does not exist" Jul 12 17:30:49.859140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.859101 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb"] Jul 12 17:30:49.862286 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:49.862266 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-99b987c8-openshift-default-b7b89b7c9-9lpkb"] Jul 12 17:30:50.470752 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:50.470716 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:51.040101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:51.040060 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd02b732-e938-4883-8638-24994555f0ea" path="/var/lib/kubelet/pods/cd02b732-e938-4883-8638-24994555f0ea/volumes" Jul 12 17:30:52.086842 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:52.086805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:52.190185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:52.190153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:52.287295 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:52.287225 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:52.387820 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:52.387736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:30:52.823516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:30:52.823486 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:18.845587 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:18.845546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:24.517141 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:24.517103 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:24.625340 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:24.625308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:24.717656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:24.717624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:40.739933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:40.739798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:46.398437 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:46.398389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:46.457435 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:46.457372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:46.544696 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:46.544650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:46.637318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:46.637282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:57.166968 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:57.166865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:31:58.080422 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:31:58.080389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:03.871218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:03.871185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:03.942212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:03.942179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:04.013892 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:04.013856 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:04.114840 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:04.114808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:04.215256 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:04.215210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:05.019720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:05.019687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:05.123004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:05.122967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:05.221776 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:05.221744 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:07.044508 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:07.044456 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:12.786897 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:12.786862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:12.884816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:12.884784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:13.550692 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:13.550654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:19.604319 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:19.604283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:19.660275 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:19.660235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:19.844696 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:19.844652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:20.141991 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.141951 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d"] Jul 12 17:32:20.142432 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142417 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" Jul 12 17:32:20.142492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142434 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" Jul 12 17:32:20.142492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142444 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" Jul 12 17:32:20.142492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142450 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" Jul 12 17:32:20.142492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142463 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d831443-7f9e-4718-8d60-3b7fbfd2eefa" containerName="maas-api" Jul 12 17:32:20.142492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142469 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d831443-7f9e-4718-8d60-3b7fbfd2eefa" containerName="maas-api" Jul 12 17:32:20.142645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142537 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d831443-7f9e-4718-8d60-3b7fbfd2eefa" containerName="maas-api" Jul 12 17:32:20.142645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.142548 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" Jul 12 17:32:20.145772 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.145755 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.148483 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.148461 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 17:32:20.148586 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.148461 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-24c431b4-openshift-default-dockercfg-t7957\"" Jul 12 17:32:20.163564 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.163539 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d"] Jul 12 17:32:20.165073 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.165047 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:20.265962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.265900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-credential-socket\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-data\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-certs\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266124 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-token\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266237 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istiod-ca-cert\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266237 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-podinfo\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-envoy\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266345 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sk9\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-kube-api-access-54sk9\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.266389 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.266371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-socket\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.367520 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-credential-socket\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.367712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-data\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.367712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-certs\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.367712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-token\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.367712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istiod-ca-cert\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.367712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-podinfo\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-envoy\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sk9\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-kube-api-access-54sk9\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-socket\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.367998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-credential-socket\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.368013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-certs\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.368123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-data\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.368219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-socket\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.368423 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.368406 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istiod-ca-cert\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.370048 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.370025 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-envoy\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.370311 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.370295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-podinfo\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.375403 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.375380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-token\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.375728 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.375705 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sk9\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-kube-api-access-54sk9\") pod \"e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.457714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.457673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:20.521641 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.521601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:20.609248 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.609220 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d"] Jul 12 17:32:20.611337 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:32:20.611295 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083531c4_a0e4_4a55_af24_2d5e5ba2b125.slice/crio-2ba8cfaa429fc86218770dcfb01e1b65a13392a78b19ce82d580ced1a53dbd40 WatchSource:0}: Error finding container 2ba8cfaa429fc86218770dcfb01e1b65a13392a78b19ce82d580ced1a53dbd40: Status 404 returned error can't find the container with id 2ba8cfaa429fc86218770dcfb01e1b65a13392a78b19ce82d580ced1a53dbd40 Jul 12 17:32:20.613516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.613488 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:32:20.613628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.613554 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:32:20.613628 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:20.613581 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:32:21.328979 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:21.328941 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" event={"ID":"083531c4-a0e4-4a55-af24-2d5e5ba2b125","Type":"ContainerStarted","Data":"b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a"} Jul 12 17:32:21.329173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:21.329005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" event={"ID":"083531c4-a0e4-4a55-af24-2d5e5ba2b125","Type":"ContainerStarted","Data":"2ba8cfaa429fc86218770dcfb01e1b65a13392a78b19ce82d580ced1a53dbd40"} Jul 12 17:32:21.352931 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:21.352865 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podStartSLOduration=1.352847455 podStartE2EDuration="1.352847455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:32:21.350385301 +0000 UTC m=+1748.845291775" watchObservedRunningTime="2026-07-12 17:32:21.352847455 +0000 UTC m=+1748.847753894" Jul 12 17:32:21.458618 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:21.458587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:21.459999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:21.459967 2573 patch_prober.go:28] interesting pod/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 12 17:32:21.460126 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:21.460026 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 12 17:32:22.461573 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:22.461536 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:23.312395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.312356 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:23.340017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.339981 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:23.341311 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.341287 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:23.387064 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.387016 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:23.453598 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.453567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:23.657329 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.657299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:23.950319 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:23.950221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:24.051936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:24.051854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:27.862106 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:27.862075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:28.574308 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:28.574278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:28.670822 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:28.670787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:29.045397 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:29.045363 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:34.720371 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:34.720329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:34.784084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:34.784046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:37.418282 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:37.418249 2573 generic.go:356] "Generic (PLEG): container finished" podID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerID="b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a" exitCode=0 Jul 12 17:32:37.418765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:37.418331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" event={"ID":"083531c4-a0e4-4a55-af24-2d5e5ba2b125","Type":"ContainerDied","Data":"b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a"} Jul 12 17:32:37.418814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:37.418790 2573 scope.go:117] "RemoveContainer" containerID="b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a" Jul 12 17:32:37.419783 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:37.419759 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:32:37.419833 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:37.419823 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:32:37.419871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:37.419862 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:32:38.340323 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:38.340272 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:38.426018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:38.425986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" event={"ID":"083531c4-a0e4-4a55-af24-2d5e5ba2b125","Type":"ContainerStarted","Data":"4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4"} Jul 12 17:32:38.458316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:38.458272 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:38.459734 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:38.459696 2573 patch_prober.go:28] interesting pod/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 12 17:32:38.459887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:38.459768 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 12 17:32:39.458398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:39.458338 2573 patch_prober.go:28] interesting pod/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 12 17:32:39.458898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:39.458442 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 12 17:32:40.458141 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:40.458101 2573 patch_prober.go:28] interesting pod/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 12 17:32:40.458357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:40.458175 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 12 17:32:41.459201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:41.459151 2573 patch_prober.go:28] interesting pod/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 12 17:32:41.459638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:41.459243 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 12 17:32:42.458234 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:42.458192 2573 patch_prober.go:28] interesting pod/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 12 17:32:42.458440 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:42.458286 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 12 17:32:43.465220 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:43.465186 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:43.465765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:43.465607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:43.466489 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:43.466470 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:32:44.117474 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:44.117435 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:44.218511 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:44.218478 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:45.344671 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:45.344634 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:50.187226 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:50.187181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:55.820971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:55.820940 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:32:55.886583 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:32:55.886524 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:00.901931 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:00.901877 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d"] Jul 12 17:33:00.902365 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:00.902161 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" containerID="cri-o://4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4" gracePeriod=30 Jul 12 17:33:00.917188 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:00.917160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:01.161486 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:01.161447 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:01.301269 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:01.301239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:01.402164 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:01.402129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:01.524213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:01.524129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:06.162857 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.162834 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:33:06.312765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312683 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-credential-socket\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.312765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312725 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-data\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.312765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312754 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istiod-ca-cert\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312793 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-token\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312831 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54sk9\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-kube-api-access-54sk9\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312861 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-envoy\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312888 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-certs\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312957 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-podinfo\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.312989 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-socket\") pod \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\" (UID: \"083531c4-a0e4-4a55-af24-2d5e5ba2b125\") " Jul 12 17:33:06.313086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313025 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:33:06.313475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313249 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:33:06.313475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313250 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:33:06.313475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313377 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.313475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313397 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.313475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313411 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.313475 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313438 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:33:06.313798 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.313555 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-data" (OuterVolumeSpecName: "istio-data") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:33:06.315094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.315069 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-token" (OuterVolumeSpecName: "istio-token") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:33:06.315196 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.315131 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:33:06.315196 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.315129 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-kube-api-access-54sk9" (OuterVolumeSpecName: "kube-api-access-54sk9") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "kube-api-access-54sk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:33:06.315196 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.315144 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "083531c4-a0e4-4a55-af24-2d5e5ba2b125" (UID: "083531c4-a0e4-4a55-af24-2d5e5ba2b125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:33:06.414400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.414368 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.414400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.414394 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.414400 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.414405 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sk9\" (UniqueName: \"kubernetes.io/projected/083531c4-a0e4-4a55-af24-2d5e5ba2b125-kube-api-access-54sk9\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.414638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.414415 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.414638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.414424 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083531c4-a0e4-4a55-af24-2d5e5ba2b125-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.414638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.414431 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083531c4-a0e4-4a55-af24-2d5e5ba2b125-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:33:06.595357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.595264 2573 generic.go:356] "Generic (PLEG): container finished" podID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerID="4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4" exitCode=0 Jul 12 17:33:06.595357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.595309 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" Jul 12 17:33:06.595357 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.595340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" event={"ID":"083531c4-a0e4-4a55-af24-2d5e5ba2b125","Type":"ContainerDied","Data":"4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4"} Jul 12 17:33:06.595595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.595387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d" event={"ID":"083531c4-a0e4-4a55-af24-2d5e5ba2b125","Type":"ContainerDied","Data":"2ba8cfaa429fc86218770dcfb01e1b65a13392a78b19ce82d580ced1a53dbd40"} Jul 12 17:33:06.595595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.595410 2573 scope.go:117] "RemoveContainer" containerID="4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4" Jul 12 17:33:06.605356 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.605341 2573 scope.go:117] "RemoveContainer" containerID="b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a" Jul 12 17:33:06.613992 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.613961 2573 scope.go:117] "RemoveContainer" containerID="4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4" Jul 12 17:33:06.614367 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:33:06.614340 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4\": container with ID starting with 4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4 not found: ID does not exist" containerID="4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4" Jul 12 17:33:06.614453 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.614378 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4"} err="failed to get container status \"4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4\": rpc error: code = NotFound desc = could not find container \"4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4\": container with ID starting with 4fecdd1b522df05ad52e87a892ea735788cab7cfdbab9279a7da79be6a5d96b4 not found: ID does not exist" Jul 12 17:33:06.614453 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.614401 2573 scope.go:117] "RemoveContainer" containerID="b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a" Jul 12 17:33:06.614651 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:33:06.614633 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a\": container with ID starting with b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a not found: ID does not exist" containerID="b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a" Jul 12 17:33:06.614713 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.614661 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a"} err="failed to get container status \"b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a\": rpc error: code = NotFound desc = could not find container \"b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a\": container with ID starting with b712b51f30e5c2d6d4c26cfbbdcac4739d39b758bc39dba3e62ebb05f2b62c9a not found: ID does not exist" Jul 12 17:33:06.616617 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.616594 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d"] Jul 12 17:33:06.621819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:06.621797 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-24c431b4-openshift-default-784bd779b7-hl69d"] Jul 12 17:33:07.039219 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:07.039183 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" path="/var/lib/kubelet/pods/083531c4-a0e4-4a55-af24-2d5e5ba2b125/volumes" Jul 12 17:33:07.221014 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:07.220983 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:07.275515 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:07.275484 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:07.357612 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:07.357491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:07.457101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:07.457068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:07.611213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:07.611127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:13.201111 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:13.201080 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:13.265143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:13.265104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:13.526581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:13.526498 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:33:13.532700 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:13.532680 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:33:20.154043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.154012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:20.453046 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.452943 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr"] Jul 12 17:33:20.453597 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453566 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" Jul 12 17:33:20.453597 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453591 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" Jul 12 17:33:20.453800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453608 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" Jul 12 17:33:20.453800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453617 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" Jul 12 17:33:20.453800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453713 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd02b732-e938-4883-8638-24994555f0ea" containerName="istio-proxy" Jul 12 17:33:20.453800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453728 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" Jul 12 17:33:20.453800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.453736 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="083531c4-a0e4-4a55-af24-2d5e5ba2b125" containerName="istio-proxy" Jul 12 17:33:20.457006 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.456989 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.459747 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.459723 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 17:33:20.459845 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.459801 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d91d15cb-openshift-default-dockercfg-9krcm\"" Jul 12 17:33:20.467783 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.467758 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr"] Jul 12 17:33:20.479516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.479474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:20.548674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81750da1-002e-4d4e-9481-2b46428e5efc-istiod-ca-cert\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.548674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-credential-socket\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.548894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-data\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.548894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548730 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-socket\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.548894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7xm\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-kube-api-access-ph7xm\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.548894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-certs\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.549138 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.548934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-istio-token\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.549138 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.549029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-envoy\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.549138 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.549073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81750da1-002e-4d4e-9481-2b46428e5efc-istio-podinfo\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.650559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-istio-token\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.650756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-envoy\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.650756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81750da1-002e-4d4e-9481-2b46428e5efc-istio-podinfo\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.650756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81750da1-002e-4d4e-9481-2b46428e5efc-istiod-ca-cert\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.650756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-credential-socket\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-data\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-socket\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7xm\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-kube-api-access-ph7xm\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.650971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-certs\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651263 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.651161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-credential-socket\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651263 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.651254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-data\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.651275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-socket\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651437 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.651385 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-certs\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.651497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.651485 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81750da1-002e-4d4e-9481-2b46428e5efc-istiod-ca-cert\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.652826 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.652800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-envoy\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.653093 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.653077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81750da1-002e-4d4e-9481-2b46428e5efc-istio-podinfo\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.669214 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.669186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-istio-token\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.669388 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.669368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7xm\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-kube-api-access-ph7xm\") pod \"e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.681375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.681348 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:20.769160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.769131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:20.906051 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.906028 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr"] Jul 12 17:33:20.907966 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:33:20.907943 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81750da1_002e_4d4e_9481_2b46428e5efc.slice/crio-9fe6d4761738dc9943a66020a1ad02db463076c32892eb8da85fbff51b7dbaab WatchSource:0}: Error finding container 9fe6d4761738dc9943a66020a1ad02db463076c32892eb8da85fbff51b7dbaab: Status 404 returned error can't find the container with id 9fe6d4761738dc9943a66020a1ad02db463076c32892eb8da85fbff51b7dbaab Jul 12 17:33:20.909993 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.909961 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:33:20.910089 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.910026 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:33:20.910089 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:20.910057 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:33:21.682457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:21.682425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" event={"ID":"81750da1-002e-4d4e-9481-2b46428e5efc","Type":"ContainerStarted","Data":"709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc"} Jul 12 17:33:21.682944 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:21.682483 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" event={"ID":"81750da1-002e-4d4e-9481-2b46428e5efc","Type":"ContainerStarted","Data":"9fe6d4761738dc9943a66020a1ad02db463076c32892eb8da85fbff51b7dbaab"} Jul 12 17:33:21.702190 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:21.702146 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" podStartSLOduration=1.702129926 podStartE2EDuration="1.702129926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:33:21.70031429 +0000 UTC m=+1809.195220757" watchObservedRunningTime="2026-07-12 17:33:21.702129926 +0000 UTC m=+1809.197036366" Jul 12 17:33:21.769936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:21.769880 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:21.774765 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:21.774738 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:22.687562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:22.687531 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:22.688650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:22.688632 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:23.645701 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:23.645664 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:23.733356 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:23.733323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:23.798437 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:23.798401 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:24.224128 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:24.224091 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:24.325491 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:24.325459 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:24.424878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:24.424842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:28.222091 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:28.222058 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:37.689683 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.689643 2573 patch_prober.go:28] interesting pod/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 12 17:33:37.690143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.689700 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 12 17:33:37.781771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.781741 2573 generic.go:356] "Generic (PLEG): container finished" podID="81750da1-002e-4d4e-9481-2b46428e5efc" containerID="709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc" exitCode=0 Jul 12 17:33:37.781974 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.781778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" event={"ID":"81750da1-002e-4d4e-9481-2b46428e5efc","Type":"ContainerDied","Data":"709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc"} Jul 12 17:33:37.782217 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.782204 2573 scope.go:117] "RemoveContainer" containerID="709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc" Jul 12 17:33:37.783223 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.783192 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:33:37.783313 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.783292 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:33:37.783350 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:37.783334 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:33:38.614962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:38.614904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:33:38.788927 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:38.788882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" event={"ID":"81750da1-002e-4d4e-9481-2b46428e5efc","Type":"ContainerStarted","Data":"8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774"} Jul 12 17:33:39.769832 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:39.769797 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:39.770092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:39.770066 2573 patch_prober.go:28] interesting pod/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 12 17:33:39.770140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:39.770118 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 12 17:33:40.770479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:40.770435 2573 patch_prober.go:28] interesting pod/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 12 17:33:40.770896 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:40.770507 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 12 17:33:41.769807 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:41.769766 2573 patch_prober.go:28] interesting pod/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 12 17:33:41.770052 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:41.769826 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 12 17:33:42.773599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:42.773565 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:42.774062 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:42.773941 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:33:42.774656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:33:42.774634 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:34:13.187194 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:34:13.187161 2573 scope.go:117] "RemoveContainer" containerID="3ff4de071952078b68bc9e6b0cc45600141f32368bfabadf745b7889bd4b153c" Jul 12 17:34:41.083882 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:34:41.083789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:35:11.804495 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:35:11.804457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:35:11.906521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:35:11.906486 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:37:11.315079 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:37:11.315038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:37:12.632856 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:37:12.632818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:37:21.185622 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:37:21.185558 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:37:25.877284 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:37:25.877236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:37:30.955040 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:37:30.955001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:06.849870 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:06.849778 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:07.050450 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:07.050417 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:08.065671 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:08.065634 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:13.570626 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:13.570594 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:38:13.578339 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:13.578317 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:38:13.583013 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:13.582985 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:13.643559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:13.643531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:16.660868 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:16.660823 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr"] Jul 12 17:38:16.661365 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:16.661070 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" containerID="cri-o://8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774" gracePeriod=30 Jul 12 17:38:16.677283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:16.677249 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:17.089687 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:17.089596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:18.111084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:18.111050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:18.212531 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:18.212494 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:18.312645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:18.312598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:18.413663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:18.413636 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:18.513312 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:18.513284 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:19.248766 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:19.248730 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:19.348039 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:19.347986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:19.454326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:19.454287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:20.970094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:20.970053 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:21.930577 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:21.930513 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:38:22.040232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040198 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-socket\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040239 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-istio-token\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040301 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-credential-socket\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040337 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-envoy\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040364 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81750da1-002e-4d4e-9481-2b46428e5efc-istio-podinfo\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040403 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81750da1-002e-4d4e-9481-2b46428e5efc-istiod-ca-cert\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040438 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-certs\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040653 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040522 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-data\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.040731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040710 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:38:22.041253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040748 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph7xm\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-kube-api-access-ph7xm\") pod \"81750da1-002e-4d4e-9481-2b46428e5efc\" (UID: \"81750da1-002e-4d4e-9481-2b46428e5efc\") " Jul 12 17:38:22.041253 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.040777 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81750da1-002e-4d4e-9481-2b46428e5efc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:38:22.041374 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.041334 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81750da1-002e-4d4e-9481-2b46428e5efc-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.041374 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.041352 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.041374 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.041364 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.041543 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.041379 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.041543 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.041444 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-data" (OuterVolumeSpecName: "istio-data") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:38:22.042675 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.042644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:38:22.042874 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.042857 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-istio-token" (OuterVolumeSpecName: "istio-token") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:38:22.043016 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.042994 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-kube-api-access-ph7xm" (OuterVolumeSpecName: "kube-api-access-ph7xm") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "kube-api-access-ph7xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:38:22.043084 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.042992 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81750da1-002e-4d4e-9481-2b46428e5efc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81750da1-002e-4d4e-9481-2b46428e5efc" (UID: "81750da1-002e-4d4e-9481-2b46428e5efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:38:22.142725 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.142660 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.142725 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.142695 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81750da1-002e-4d4e-9481-2b46428e5efc-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.143015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.142846 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81750da1-002e-4d4e-9481-2b46428e5efc-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.143015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.142872 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7xm\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-kube-api-access-ph7xm\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.143015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.142889 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81750da1-002e-4d4e-9481-2b46428e5efc-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:38:22.352240 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.352138 2573 generic.go:356] "Generic (PLEG): container finished" podID="81750da1-002e-4d4e-9481-2b46428e5efc" containerID="8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774" exitCode=0 Jul 12 17:38:22.352240 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.352208 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" Jul 12 17:38:22.352240 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.352224 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" event={"ID":"81750da1-002e-4d4e-9481-2b46428e5efc","Type":"ContainerDied","Data":"8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774"} Jul 12 17:38:22.352591 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.352292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr" event={"ID":"81750da1-002e-4d4e-9481-2b46428e5efc","Type":"ContainerDied","Data":"9fe6d4761738dc9943a66020a1ad02db463076c32892eb8da85fbff51b7dbaab"} Jul 12 17:38:22.352591 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.352323 2573 scope.go:117] "RemoveContainer" containerID="8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774" Jul 12 17:38:22.362366 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.362345 2573 scope.go:117] "RemoveContainer" containerID="709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc" Jul 12 17:38:22.372193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.372174 2573 scope.go:117] "RemoveContainer" containerID="8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774" Jul 12 17:38:22.372490 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:38:22.372466 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774\": container with ID starting with 8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774 not found: ID does not exist" containerID="8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774" Jul 12 17:38:22.372557 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.372503 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774"} err="failed to get container status \"8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774\": rpc error: code = NotFound desc = could not find container \"8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774\": container with ID starting with 8dc612909ea5baf4e84e3211baa738d10ffb0b927806bde86a21d60bf77bb774 not found: ID does not exist" Jul 12 17:38:22.372557 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.372529 2573 scope.go:117] "RemoveContainer" containerID="709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc" Jul 12 17:38:22.372837 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:38:22.372812 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc\": container with ID starting with 709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc not found: ID does not exist" containerID="709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc" Jul 12 17:38:22.372902 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.372846 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc"} err="failed to get container status \"709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc\": rpc error: code = NotFound desc = could not find container \"709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc\": container with ID starting with 709f08d76d16cb9e33a3be645958e2d5ea1653da96ee401e775348a3f692e9fc not found: ID does not exist" Jul 12 17:38:22.379368 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.379343 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr"] Jul 12 17:38:22.387560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:22.387539 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d91d15cb-openshift-default-6f97d5cb55-q7srr"] Jul 12 17:38:23.042061 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:23.042027 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" path="/var/lib/kubelet/pods/81750da1-002e-4d4e-9481-2b46428e5efc/volumes" Jul 12 17:38:26.765801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:26.765769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:26.830462 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:26.830425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:27.440865 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:27.440826 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:33.175743 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:33.175694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:33.231966 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:33.231896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:33.316311 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:33.316272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:33.414888 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:33.414850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:39.062245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:39.062207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:39.120663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:39.120632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:43.917052 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:43.917012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:38:44.301133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:38:44.301044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:05.283213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:05.283129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:10.883395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:10.883362 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:10.983656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:10.983616 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:11.254000 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.253962 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:11.549521 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.549447 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp"] Jul 12 17:39:11.549922 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.549898 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" Jul 12 17:39:11.549981 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.549928 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" Jul 12 17:39:11.549981 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.549950 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" Jul 12 17:39:11.549981 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.549955 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" Jul 12 17:39:11.550080 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.550028 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" Jul 12 17:39:11.550080 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.550038 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81750da1-002e-4d4e-9481-2b46428e5efc" containerName="istio-proxy" Jul 12 17:39:11.553313 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.553296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.555998 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.555974 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 17:39:11.556109 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.556068 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c5fd66-openshift-default-dockercfg-cn24v\"" Jul 12 17:39:11.563679 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.563655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp"] Jul 12 17:39:11.571465 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.571437 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:11.713532 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-istio-token\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.713730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-envoy\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.713730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-credential-socket\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.713730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmxd\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-kube-api-access-kcmxd\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.713730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25136958-76a2-4d65-9a13-7915c67f66e1-istio-podinfo\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.713730 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-certs\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.714121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25136958-76a2-4d65-9a13-7915c67f66e1-istiod-ca-cert\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.714121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-data\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.714121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.713797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-socket\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.758145 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.758114 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:11.814989 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.814881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-istio-token\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.814989 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.814943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-envoy\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.814995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-credential-socket\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmxd\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-kube-api-access-kcmxd\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25136958-76a2-4d65-9a13-7915c67f66e1-istio-podinfo\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815236 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-certs\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25136958-76a2-4d65-9a13-7915c67f66e1-istiod-ca-cert\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-data\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815479 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-socket\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815635 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-credential-socket\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815635 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815591 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-certs\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-data\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.815951 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-socket\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.816096 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.815982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25136958-76a2-4d65-9a13-7915c67f66e1-istiod-ca-cert\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.817332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.817305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-envoy\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.817613 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.817596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25136958-76a2-4d65-9a13-7915c67f66e1-istio-podinfo\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.822630 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.822602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-istio-token\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.822737 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.822719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmxd\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-kube-api-access-kcmxd\") pod \"e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:11.865613 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:11.865582 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:12.004477 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.004446 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp"] Jul 12 17:39:12.006176 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:39:12.006142 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25136958_76a2_4d65_9a13_7915c67f66e1.slice/crio-dd6cc158b1c7617d5897ff2d7d85626dd61355cd89f5e74434e0d76405ac22a6 WatchSource:0}: Error finding container dd6cc158b1c7617d5897ff2d7d85626dd61355cd89f5e74434e0d76405ac22a6: Status 404 returned error can't find the container with id dd6cc158b1c7617d5897ff2d7d85626dd61355cd89f5e74434e0d76405ac22a6 Jul 12 17:39:12.008516 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.008476 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:12.008598 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.008564 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:12.008639 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.008607 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:12.627438 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.627403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" event={"ID":"25136958-76a2-4d65-9a13-7915c67f66e1","Type":"ContainerStarted","Data":"aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7"} Jul 12 17:39:12.627618 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.627458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" event={"ID":"25136958-76a2-4d65-9a13-7915c67f66e1","Type":"ContainerStarted","Data":"dd6cc158b1c7617d5897ff2d7d85626dd61355cd89f5e74434e0d76405ac22a6"} Jul 12 17:39:12.648520 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.648474 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podStartSLOduration=1.648459549 podStartE2EDuration="1.648459549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:39:12.645766889 +0000 UTC m=+2160.140673328" watchObservedRunningTime="2026-07-12 17:39:12.648459549 +0000 UTC m=+2160.143365989" Jul 12 17:39:12.865803 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:12.865776 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:13.871174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:13.871148 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:14.637673 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:14.637637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:14.640295 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:14.640268 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:14.641352 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:14.641315 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:14.739892 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:14.739853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:14.916335 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:14.916303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:15.279263 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:15.279155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:15.383478 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:15.383420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:15.485463 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:15.485426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:19.236029 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.235986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:19.604229 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.604141 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:19.911747 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.911718 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4"] Jul 12 17:39:19.915358 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.915340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:19.917867 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.917843 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7fa4a5-openshift-default-dockercfg-cnpnf\"" Jul 12 17:39:19.925433 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.925411 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4"] Jul 12 17:39:19.934062 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:19.934033 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:20.096186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-token\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096190 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-credential-socket\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-envoy\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-podinfo\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istiod-ca-cert\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-certs\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096534 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghgn\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-kube-api-access-7ghgn\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096534 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-socket\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.096534 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.096474 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-data\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197285 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-data\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197458 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-token\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197458 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-credential-socket\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197458 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-envoy\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197458 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-podinfo\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197458 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istiod-ca-cert\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197458 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-certs\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghgn\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-kube-api-access-7ghgn\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-socket\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-data\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.197784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-credential-socket\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.198044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-socket\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.198044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.197969 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-certs\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.198284 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.198265 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istiod-ca-cert\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.199649 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.199628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-envoy\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.200028 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.200010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-podinfo\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.205134 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.205105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghgn\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-kube-api-access-7ghgn\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.205214 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.205183 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-token\") pod \"e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.226853 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.226833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:20.369751 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.369629 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4"] Jul 12 17:39:20.372163 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:39:20.372137 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9ea00f_47b5_419b_9a1c_24b0f21fd2ea.slice/crio-d6917f73496f675d47984982e23d5d45499a7a984972e93db5a73ec946b289fd WatchSource:0}: Error finding container d6917f73496f675d47984982e23d5d45499a7a984972e93db5a73ec946b289fd: Status 404 returned error can't find the container with id d6917f73496f675d47984982e23d5d45499a7a984972e93db5a73ec946b289fd Jul 12 17:39:20.374431 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.374401 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:20.374517 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.374464 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:20.374517 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.374494 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:20.675233 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.675197 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" event={"ID":"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea","Type":"ContainerStarted","Data":"89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca"} Jul 12 17:39:20.675412 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.675271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" event={"ID":"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea","Type":"ContainerStarted","Data":"d6917f73496f675d47984982e23d5d45499a7a984972e93db5a73ec946b289fd"} Jul 12 17:39:20.696012 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.695962 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podStartSLOduration=1.695945086 podStartE2EDuration="1.695945086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:39:20.69382318 +0000 UTC m=+2168.188729619" watchObservedRunningTime="2026-07-12 17:39:20.695945086 +0000 UTC m=+2168.190851525" Jul 12 17:39:20.877840 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:20.877805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:21.227573 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:21.227537 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:21.232750 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:21.232726 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:21.681418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:21.681389 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:21.682492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:21.682475 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:27.670513 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:27.670466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:27.772879 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:27.772837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:27.847898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:27.847855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:27.938950 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:27.938831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:28.306721 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:28.306687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:28.408239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:28.408205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:30.681121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:30.681079 2573 patch_prober.go:28] interesting pod/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 12 17:39:30.681515 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:30.681149 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 12 17:39:31.739293 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:31.739250 2573 generic.go:356] "Generic (PLEG): container finished" podID="25136958-76a2-4d65-9a13-7915c67f66e1" containerID="aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7" exitCode=0 Jul 12 17:39:31.739774 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:31.739333 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" event={"ID":"25136958-76a2-4d65-9a13-7915c67f66e1","Type":"ContainerDied","Data":"aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7"} Jul 12 17:39:31.739774 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:31.739739 2573 scope.go:117] "RemoveContainer" containerID="aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7" Jul 12 17:39:31.740795 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:31.740767 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:31.740865 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:31.740849 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:31.740924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:31.740897 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:32.218193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:32.218156 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:32.751450 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:32.751407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" event={"ID":"25136958-76a2-4d65-9a13-7915c67f66e1","Type":"ContainerStarted","Data":"81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7"} Jul 12 17:39:32.866266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:32.866228 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:32.866472 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:32.866358 2573 patch_prober.go:28] interesting pod/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 12 17:39:32.866472 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:32.866421 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 12 17:39:33.866721 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:33.866677 2573 patch_prober.go:28] interesting pod/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 12 17:39:33.867243 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:33.866746 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 12 17:39:34.866551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:34.866503 2573 patch_prober.go:28] interesting pod/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 12 17:39:34.866769 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:34.866591 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 12 17:39:35.866954 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:35.866897 2573 patch_prober.go:28] interesting pod/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 12 17:39:35.867562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:35.867528 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 12 17:39:36.136464 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:36.136362 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:36.236884 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:36.236840 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:36.869696 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:36.869604 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:36.870155 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:36.870038 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:36.870673 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:36.870653 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:39:37.723103 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:37.723065 2573 patch_prober.go:28] interesting pod/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 12 17:39:37.723305 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:37.723125 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 12 17:39:39.801993 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:39.801892 2573 generic.go:356] "Generic (PLEG): container finished" podID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerID="89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca" exitCode=0 Jul 12 17:39:39.801993 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:39.801971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" event={"ID":"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea","Type":"ContainerDied","Data":"89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca"} Jul 12 17:39:39.802570 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:39.802485 2573 scope.go:117] "RemoveContainer" containerID="89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca" Jul 12 17:39:39.803529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:39.803504 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:39.803631 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:39.803585 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:39.803694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:39.803628 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:39:40.810006 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:40.809971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" event={"ID":"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea","Type":"ContainerStarted","Data":"c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00"} Jul 12 17:39:41.227638 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:41.227603 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:41.227858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:41.227789 2573 patch_prober.go:28] interesting pod/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 12 17:39:41.227858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:41.227831 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 12 17:39:42.227388 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:42.227348 2573 patch_prober.go:28] interesting pod/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 12 17:39:42.227872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:42.227414 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 12 17:39:43.228182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:43.228144 2573 patch_prober.go:28] interesting pod/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 12 17:39:43.228566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:43.228199 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 12 17:39:44.227686 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:44.227646 2573 patch_prober.go:28] interesting pod/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 12 17:39:44.227878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:44.227705 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 12 17:39:45.230590 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:45.230555 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:45.231058 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:45.230839 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:45.231510 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:45.231494 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:39:49.050443 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:49.050407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:49.152267 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:49.152229 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:39:50.018485 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:39:50.018443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:12.580457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:12.580419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:12.682206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:12.682177 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:15.928171 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:15.928130 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:36.442255 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:36.442172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:37.528681 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:37.528647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:38.093671 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:38.093634 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:38.191222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:38.191186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:48.016562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:48.016521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:52.707713 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:52.707679 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:58.679383 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:58.679314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:40:58.737342 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:40:58.737303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:03.370494 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:03.370458 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp"] Jul 12 17:41:03.370988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:03.370691 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" containerID="cri-o://81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7" gracePeriod=30 Jul 12 17:41:03.398451 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:03.398379 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:03.634419 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:03.634297 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:04.206182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:04.206150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:04.307854 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:04.307825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:06.872564 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:06.872525 2573 patch_prober.go:28] interesting pod/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 17:41:06.872976 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:06.872599 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 17:41:08.627738 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.627714 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:41:08.782006 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.781971 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-socket\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782023 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcmxd\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-kube-api-access-kcmxd\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782041 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25136958-76a2-4d65-9a13-7915c67f66e1-istiod-ca-cert\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782213 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782176 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25136958-76a2-4d65-9a13-7915c67f66e1-istio-podinfo\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782238 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-credential-socket\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782283 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:08.782395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782305 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-data\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-istio-token\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782395 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782393 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-certs\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782436 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-envoy\") pod \"25136958-76a2-4d65-9a13-7915c67f66e1\" (UID: \"25136958-76a2-4d65-9a13-7915c67f66e1\") " Jul 12 17:41:08.782656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782539 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:08.782656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782592 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25136958-76a2-4d65-9a13-7915c67f66e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:41:08.782817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782801 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.782878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782820 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25136958-76a2-4d65-9a13-7915c67f66e1-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.782878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.782835 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.783301 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.783272 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:08.783399 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.783345 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:08.784398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.784366 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25136958-76a2-4d65-9a13-7915c67f66e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:41:08.784507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.784411 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:41:08.784507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.784432 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-kube-api-access-kcmxd" (OuterVolumeSpecName: "kube-api-access-kcmxd") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "kube-api-access-kcmxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:41:08.784507 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.784492 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25136958-76a2-4d65-9a13-7915c67f66e1" (UID: "25136958-76a2-4d65-9a13-7915c67f66e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:08.884320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.884286 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcmxd\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-kube-api-access-kcmxd\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.884320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.884315 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25136958-76a2-4d65-9a13-7915c67f66e1-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.884320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.884326 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.884561 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.884334 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25136958-76a2-4d65-9a13-7915c67f66e1-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.884561 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.884342 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:08.884561 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:08.884349 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25136958-76a2-4d65-9a13-7915c67f66e1-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:09.298971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.298863 2573 generic.go:356] "Generic (PLEG): container finished" podID="25136958-76a2-4d65-9a13-7915c67f66e1" containerID="81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7" exitCode=0 Jul 12 17:41:09.298971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.298937 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" Jul 12 17:41:09.298971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.298946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" event={"ID":"25136958-76a2-4d65-9a13-7915c67f66e1","Type":"ContainerDied","Data":"81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7"} Jul 12 17:41:09.299201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.299004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp" event={"ID":"25136958-76a2-4d65-9a13-7915c67f66e1","Type":"ContainerDied","Data":"dd6cc158b1c7617d5897ff2d7d85626dd61355cd89f5e74434e0d76405ac22a6"} Jul 12 17:41:09.299201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.299026 2573 scope.go:117] "RemoveContainer" containerID="81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7" Jul 12 17:41:09.308042 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.308023 2573 scope.go:117] "RemoveContainer" containerID="aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7" Jul 12 17:41:09.316518 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.316499 2573 scope.go:117] "RemoveContainer" containerID="81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7" Jul 12 17:41:09.316786 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:41:09.316763 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7\": container with ID starting with 81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7 not found: ID does not exist" containerID="81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7" Jul 12 17:41:09.316880 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.316795 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7"} err="failed to get container status \"81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7\": rpc error: code = NotFound desc = could not find container \"81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7\": container with ID starting with 81e282c09c04b3d02168c9a4c12be274543935d9b12382f3fb71f06f0b08abc7 not found: ID does not exist" Jul 12 17:41:09.316880 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.316816 2573 scope.go:117] "RemoveContainer" containerID="aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7" Jul 12 17:41:09.317082 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:41:09.317065 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7\": container with ID starting with aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7 not found: ID does not exist" containerID="aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7" Jul 12 17:41:09.317142 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.317086 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7"} err="failed to get container status \"aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7\": rpc error: code = NotFound desc = could not find container \"aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7\": container with ID starting with aa4e2062a1e243dbde79e47f99e5c1ee81dac823e70eaa14d0904d2bab9979a7 not found: ID does not exist" Jul 12 17:41:09.318160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.318139 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp"] Jul 12 17:41:09.324276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:09.324257 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c5fd66-openshift-default-66849f9d77-9hjgp"] Jul 12 17:41:11.039720 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:11.039686 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" path="/var/lib/kubelet/pods/25136958-76a2-4d65-9a13-7915c67f66e1/volumes" Jul 12 17:41:14.311457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:14.311422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:18.854122 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:18.854083 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:24.686780 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:24.686740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:24.787378 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:24.787326 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:29.542800 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:29.542756 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4"] Jul 12 17:41:29.543666 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:29.543608 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" containerID="cri-o://c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00" gracePeriod=30 Jul 12 17:41:29.556204 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:29.556148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:29.802158 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:29.802062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:30.153544 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.153517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:30.233734 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.233691 2573 patch_prober.go:28] interesting pod/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 17:41:30.233898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.233752 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 17:41:30.462085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.461995 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g"] Jul 12 17:41:30.462557 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.462537 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" Jul 12 17:41:30.462659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.462558 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" Jul 12 17:41:30.462659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.462593 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" Jul 12 17:41:30.462659 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.462602 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" Jul 12 17:41:30.462822 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.462699 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" Jul 12 17:41:30.462822 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.462714 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="25136958-76a2-4d65-9a13-7915c67f66e1" containerName="istio-proxy" Jul 12 17:41:30.466042 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.466014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.468559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.468534 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-444d10-openshift-default-dockercfg-n7n5k\"" Jul 12 17:41:30.481151 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.481123 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g"] Jul 12 17:41:30.488635 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.488602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:30.577872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.577839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-certs\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.577872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.577869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-podinfo\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.577977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7r9b\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-kube-api-access-t7r9b\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.578061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-envoy\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.578122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-socket\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.578142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-data\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.578184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c3c64c-ca69-4527-ac43-76ebfc46a520-istiod-ca-cert\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.578219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-credential-socket\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.578283 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.578234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-token\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7r9b\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-kube-api-access-t7r9b\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-envoy\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-socket\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-data\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c3c64c-ca69-4527-ac43-76ebfc46a520-istiod-ca-cert\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-credential-socket\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-token\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679436 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-certs\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-podinfo\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-socket\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.679814 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-data\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.680019 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.679987 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-credential-socket\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.680019 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.680008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-certs\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.680480 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.680453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c3c64c-ca69-4527-ac43-76ebfc46a520-istiod-ca-cert\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.681757 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.681736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-envoy\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.681989 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.681973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-podinfo\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.690993 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.690962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-token\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.691096 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.691016 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7r9b\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-kube-api-access-t7r9b\") pod \"e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.777540 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.777450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:30.932548 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.932510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g"] Jul 12 17:41:30.937694 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:41:30.937664 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c3c64c_ca69_4527_ac43_76ebfc46a520.slice/crio-46cd110947e2fef944ede29592a4ed5622373ca61ee02a87b5437aa3d3c5b823 WatchSource:0}: Error finding container 46cd110947e2fef944ede29592a4ed5622373ca61ee02a87b5437aa3d3c5b823: Status 404 returned error can't find the container with id 46cd110947e2fef944ede29592a4ed5622373ca61ee02a87b5437aa3d3c5b823 Jul 12 17:41:30.939839 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.939812 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:30.939906 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.939878 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:30.939906 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:30.939904 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:31.029264 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:31.029176 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:31.425794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:31.425748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" event={"ID":"48c3c64c-ca69-4527-ac43-76ebfc46a520","Type":"ContainerStarted","Data":"32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d"} Jul 12 17:41:31.426030 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:31.425813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" event={"ID":"48c3c64c-ca69-4527-ac43-76ebfc46a520","Type":"ContainerStarted","Data":"46cd110947e2fef944ede29592a4ed5622373ca61ee02a87b5437aa3d3c5b823"} Jul 12 17:41:31.446437 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:31.446382 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podStartSLOduration=1.446365223 podStartE2EDuration="1.446365223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:41:31.443249136 +0000 UTC m=+2298.938155603" watchObservedRunningTime="2026-07-12 17:41:31.446365223 +0000 UTC m=+2298.941271662" Jul 12 17:41:31.778599 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:31.778517 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:32.782301 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:32.782265 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:33.392771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:33.392727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:33.437977 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:33.437947 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:33.438863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:33.438844 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:33.506045 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:33.506008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:33.644502 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:33.644401 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:33.976118 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:33.976086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:34.108137 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.108068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:34.208193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.208163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:34.809830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.809763 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:41:34.928288 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928257 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-podinfo\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928310 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-envoy\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-certs\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928423 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-token\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928497 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928460 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istiod-ca-cert\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-socket\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928529 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-credential-socket\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928554 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ghgn\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-kube-api-access-7ghgn\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928597 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-data\") pod \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\" (UID: \"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea\") " Jul 12 17:41:34.928962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928707 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:34.928962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928783 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:34.928962 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.928940 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:41:34.929148 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.929091 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:34.929148 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.929114 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:34.929148 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.929127 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:34.929433 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.929412 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:34.929493 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.929468 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:34.930594 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.930572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:41:34.930719 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.930692 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:41:34.930804 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.930785 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:41:34.931043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:34.931023 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-kube-api-access-7ghgn" (OuterVolumeSpecName: "kube-api-access-7ghgn") pod "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" (UID: "fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea"). InnerVolumeSpecName "kube-api-access-7ghgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:41:35.030106 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.030072 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:35.030106 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.030101 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:35.030106 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.030111 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghgn\" (UniqueName: \"kubernetes.io/projected/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-kube-api-access-7ghgn\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:35.030548 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.030120 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:35.030548 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.030129 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:35.030548 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.030137 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:35.454987 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.454946 2573 generic.go:356] "Generic (PLEG): container finished" podID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerID="c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00" exitCode=0 Jul 12 17:41:35.455187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.455020 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" Jul 12 17:41:35.455187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.455028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" event={"ID":"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea","Type":"ContainerDied","Data":"c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00"} Jul 12 17:41:35.455187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.455094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4" event={"ID":"fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea","Type":"ContainerDied","Data":"d6917f73496f675d47984982e23d5d45499a7a984972e93db5a73ec946b289fd"} Jul 12 17:41:35.455187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.455125 2573 scope.go:117] "RemoveContainer" containerID="c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00" Jul 12 17:41:35.465543 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.465525 2573 scope.go:117] "RemoveContainer" containerID="89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca" Jul 12 17:41:35.475162 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.475136 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4"] Jul 12 17:41:35.475510 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.475493 2573 scope.go:117] "RemoveContainer" containerID="c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00" Jul 12 17:41:35.476032 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:41:35.475998 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00\": container with ID starting with c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00 not found: ID does not exist" containerID="c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00" Jul 12 17:41:35.476032 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.476034 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00"} err="failed to get container status \"c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00\": rpc error: code = NotFound desc = could not find container \"c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00\": container with ID starting with c9050e64d70778566f448a4f67e3391ae4147da185e38530d76795dba9578d00 not found: ID does not exist" Jul 12 17:41:35.476032 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.476055 2573 scope.go:117] "RemoveContainer" containerID="89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca" Jul 12 17:41:35.476348 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:41:35.476332 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca\": container with ID starting with 89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca not found: ID does not exist" containerID="89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca" Jul 12 17:41:35.476397 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.476354 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca"} err="failed to get container status \"89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca\": rpc error: code = NotFound desc = could not find container \"89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca\": container with ID starting with 89f2d3652fe556cd141666bede20a9aed9180dc613ffc879c4d4ebff3d56a7ca not found: ID does not exist" Jul 12 17:41:35.478101 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:35.478081 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7fa4a5-openshift-default-648dc66f9-rdbx4"] Jul 12 17:41:37.040068 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:37.040032 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" path="/var/lib/kubelet/pods/fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea/volumes" Jul 12 17:41:37.924407 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:37.924373 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:38.400044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.399951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:38.681864 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.681823 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj"] Jul 12 17:41:38.682749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.682724 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" Jul 12 17:41:38.682898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.682876 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" Jul 12 17:41:38.682898 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.682901 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" Jul 12 17:41:38.683069 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.682936 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" Jul 12 17:41:38.683069 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.683044 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" Jul 12 17:41:38.683069 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.683056 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9ea00f-47b5-419b-9a1c-24b0f21fd2ea" containerName="istio-proxy" Jul 12 17:41:38.687695 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.687671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.690636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.690616 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2484ca-openshift-default-dockercfg-4w4lv\"" Jul 12 17:41:38.698200 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.698175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj"] Jul 12 17:41:38.705044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.705019 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:38.769743 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.769704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-data\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.769743 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.769743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-credential-socket\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.769768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-envoy\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.769815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-token\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.769850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-certs\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.769903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istiod-ca-cert\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770017 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.770000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-socket\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.770075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-podinfo\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.770181 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.770111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrcx\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-kube-api-access-pdrcx\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871448 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-socket\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-podinfo\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrcx\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-kube-api-access-pdrcx\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-data\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-credential-socket\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871615 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-envoy\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871904 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-token\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871904 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-certs\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871904 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istiod-ca-cert\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.871904 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871848 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-socket\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.872172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871943 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-data\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.872172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.871981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-credential-socket\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.872286 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.872219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-certs\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.872402 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.872384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istiod-ca-cert\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.874002 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.873979 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-podinfo\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.874102 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.873999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-envoy\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.879016 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.878990 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-token\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.879156 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.879139 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrcx\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-kube-api-access-pdrcx\") pod \"e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:38.998961 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:38.998836 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:39.128946 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.128889 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:39.155001 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.154977 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj"] Jul 12 17:41:39.157231 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:41:39.157197 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dd8eb0_6b57_4be3_b11d_ca519163a995.slice/crio-ede275110b3f827a6b15d96502f6ff62c3185bcc1041575ae0ab24793ea02f41 WatchSource:0}: Error finding container ede275110b3f827a6b15d96502f6ff62c3185bcc1041575ae0ab24793ea02f41: Status 404 returned error can't find the container with id ede275110b3f827a6b15d96502f6ff62c3185bcc1041575ae0ab24793ea02f41 Jul 12 17:41:39.159591 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.159550 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:39.159688 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.159637 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:39.159729 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.159685 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:39.482988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.482942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" event={"ID":"d5dd8eb0-6b57-4be3-b11d-ca519163a995","Type":"ContainerStarted","Data":"cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914"} Jul 12 17:41:39.483401 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.483065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" event={"ID":"d5dd8eb0-6b57-4be3-b11d-ca519163a995","Type":"ContainerStarted","Data":"ede275110b3f827a6b15d96502f6ff62c3185bcc1041575ae0ab24793ea02f41"} Jul 12 17:41:39.507027 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.506969 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" podStartSLOduration=1.506948629 podStartE2EDuration="1.506948629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:41:39.50614851 +0000 UTC m=+2307.001054949" watchObservedRunningTime="2026-07-12 17:41:39.506948629 +0000 UTC m=+2307.001855071" Jul 12 17:41:39.999348 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:39.999305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:41.004669 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:41.004633 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:41.494239 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:41.494201 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:41.495873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:41.495826 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:41.559298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:41.559262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:41.664853 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:41.664814 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:41.753753 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:41.753646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:42.101574 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:42.101475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:42.208470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:42.208435 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:42.309670 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:42.309639 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:45.983930 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:45.983873 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:46.522801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:46.522709 2573 generic.go:356] "Generic (PLEG): container finished" podID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerID="32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d" exitCode=0 Jul 12 17:41:46.522801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:46.522785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" event={"ID":"48c3c64c-ca69-4527-ac43-76ebfc46a520","Type":"ContainerDied","Data":"32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d"} Jul 12 17:41:46.523218 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:46.523203 2573 scope.go:117] "RemoveContainer" containerID="32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d" Jul 12 17:41:46.524267 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:46.524243 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:46.524318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:46.524306 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:46.524354 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:46.524345 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:47.529764 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:47.529729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" event={"ID":"48c3c64c-ca69-4527-ac43-76ebfc46a520","Type":"ContainerStarted","Data":"989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c"} Jul 12 17:41:47.779145 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:47.779102 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:47.779372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:47.779294 2573 patch_prober.go:28] interesting pod/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 12 17:41:47.779372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:47.779338 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 12 17:41:48.438190 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:48.438148 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:48.778936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:48.778805 2573 patch_prober.go:28] interesting pod/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 12 17:41:48.779522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:48.779485 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 12 17:41:49.778815 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:49.778773 2573 patch_prober.go:28] interesting pod/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 12 17:41:49.779211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:49.779180 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 12 17:41:50.779677 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:50.779633 2573 patch_prober.go:28] interesting pod/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 12 17:41:50.780192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:50.779705 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 12 17:41:51.784160 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:51.784132 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:51.785135 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:51.785112 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:41:54.175840 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.175804 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw"] Jul 12 17:41:54.180094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.180072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:54.182817 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.182793 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5xzcz\"" Jul 12 17:41:54.185551 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.185524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw"] Jul 12 17:41:54.330208 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.330123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70c23e-b388-4667-8b94-ab4bf69ff144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-444d10-7hmnw\" (UID: \"9a70c23e-b388-4667-8b94-ab4bf69ff144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:54.431190 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.431094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70c23e-b388-4667-8b94-ab4bf69ff144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-444d10-7hmnw\" (UID: \"9a70c23e-b388-4667-8b94-ab4bf69ff144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:54.431717 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.431698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70c23e-b388-4667-8b94-ab4bf69ff144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-444d10-7hmnw\" (UID: \"9a70c23e-b388-4667-8b94-ab4bf69ff144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:54.564606 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.564570 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:54.580700 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.580671 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerID="cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914" exitCode=0 Jul 12 17:41:54.580833 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.580746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" event={"ID":"d5dd8eb0-6b57-4be3-b11d-ca519163a995","Type":"ContainerDied","Data":"cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914"} Jul 12 17:41:54.581235 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.581220 2573 scope.go:117] "RemoveContainer" containerID="cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914" Jul 12 17:41:54.582298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.582271 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:54.582393 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.582338 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:54.582393 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.582369 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:41:54.687133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.687028 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:54.787651 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.787617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:41:54.924927 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:54.924882 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw"] Jul 12 17:41:54.927179 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:41:54.927134 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70c23e_b388_4667_8b94_ab4bf69ff144.slice/crio-4318b73bbf49b02728e0c2f05a1d29e96169e2eb31d7de999bfb144a9b40c3c3 WatchSource:0}: Error finding container 4318b73bbf49b02728e0c2f05a1d29e96169e2eb31d7de999bfb144a9b40c3c3: Status 404 returned error can't find the container with id 4318b73bbf49b02728e0c2f05a1d29e96169e2eb31d7de999bfb144a9b40c3c3 Jul 12 17:41:55.588964 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:55.588898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" event={"ID":"d5dd8eb0-6b57-4be3-b11d-ca519163a995","Type":"ContainerStarted","Data":"d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689"} Jul 12 17:41:55.590337 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:55.590309 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" event={"ID":"9a70c23e-b388-4667-8b94-ab4bf69ff144","Type":"ContainerStarted","Data":"2339334583a1cf3214d21fb26e4f95fe9402f0f43d11a3506e30ee462f5c30c5"} Jul 12 17:41:55.590488 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:55.590352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" event={"ID":"9a70c23e-b388-4667-8b94-ab4bf69ff144","Type":"ContainerStarted","Data":"4318b73bbf49b02728e0c2f05a1d29e96169e2eb31d7de999bfb144a9b40c3c3"} Jul 12 17:41:55.622419 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:55.622317 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" podStartSLOduration=1.6223009579999998 podStartE2EDuration="1.622300958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:41:55.620590378 +0000 UTC m=+2323.115496831" watchObservedRunningTime="2026-07-12 17:41:55.622300958 +0000 UTC m=+2323.117207397" Jul 12 17:41:55.999685 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:55.999650 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:56.000682 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.000658 2573 patch_prober.go:28] interesting pod/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 12 17:41:56.000753 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.000710 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 12 17:41:56.495151 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.495113 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:41:56.597288 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.597251 2573 generic.go:356] "Generic (PLEG): container finished" podID="9a70c23e-b388-4667-8b94-ab4bf69ff144" containerID="2339334583a1cf3214d21fb26e4f95fe9402f0f43d11a3506e30ee462f5c30c5" exitCode=0 Jul 12 17:41:56.597663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.597317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" event={"ID":"9a70c23e-b388-4667-8b94-ab4bf69ff144","Type":"ContainerDied","Data":"2339334583a1cf3214d21fb26e4f95fe9402f0f43d11a3506e30ee462f5c30c5"} Jul 12 17:41:56.999385 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.999348 2573 patch_prober.go:28] interesting pod/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 12 17:41:56.999567 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:56.999408 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 12 17:41:57.790271 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:57.790244 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:57.868263 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:57.868170 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70c23e-b388-4667-8b94-ab4bf69ff144-maas-api-service-ca\") pod \"9a70c23e-b388-4667-8b94-ab4bf69ff144\" (UID: \"9a70c23e-b388-4667-8b94-ab4bf69ff144\") " Jul 12 17:41:57.868540 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:57.868516 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a70c23e-b388-4667-8b94-ab4bf69ff144-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a70c23e-b388-4667-8b94-ab4bf69ff144" (UID: "9a70c23e-b388-4667-8b94-ab4bf69ff144"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:41:57.969315 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:57.969275 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70c23e-b388-4667-8b94-ab4bf69ff144-maas-api-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:41:57.999398 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:57.999355 2573 patch_prober.go:28] interesting pod/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 12 17:41:57.999566 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:57.999428 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 12 17:41:58.611763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:58.611725 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" event={"ID":"9a70c23e-b388-4667-8b94-ab4bf69ff144","Type":"ContainerDied","Data":"4318b73bbf49b02728e0c2f05a1d29e96169e2eb31d7de999bfb144a9b40c3c3"} Jul 12 17:41:58.611763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:58.611748 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-444d10-7hmnw" Jul 12 17:41:58.612085 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:58.611786 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4318b73bbf49b02728e0c2f05a1d29e96169e2eb31d7de999bfb144a9b40c3c3" Jul 12 17:41:58.999676 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:58.999631 2573 patch_prober.go:28] interesting pod/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 12 17:41:59.000174 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:41:58.999687 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 12 17:42:00.003887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:00.003848 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:42:00.004871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:00.004849 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:42:02.879924 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:02.879880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:02.981295 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:02.981254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:04.650375 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:04.650295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:04.751211 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:04.751172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:04.896742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:04.896703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:09.262830 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:09.262802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:14.950845 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:14.950808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:16.286553 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:16.286516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:19.945537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:19.945497 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g"] Jul 12 17:42:19.946029 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:19.945734 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" containerID="cri-o://989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c" gracePeriod=30 Jul 12 17:42:19.972933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:19.972146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:20.197564 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:20.197476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:20.775041 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:20.775002 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:20.834810 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:20.834763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:21.786763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:21.786728 2573 patch_prober.go:28] interesting pod/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 17:42:21.787158 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:21.786803 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 17:42:25.199030 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.199005 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:42:25.341410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341317 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-podinfo\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341367 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-certs\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341410 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-socket\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341475 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-token\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341499 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-envoy\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341541 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c3c64c-ca69-4527-ac43-76ebfc46a520-istiod-ca-cert\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341606 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-credential-socket\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341663 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341656 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-data\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.341953 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341750 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:25.341953 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341774 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:25.341953 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341945 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7r9b\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-kube-api-access-t7r9b\") pod \"48c3c64c-ca69-4527-ac43-76ebfc46a520\" (UID: \"48c3c64c-ca69-4527-ac43-76ebfc46a520\") " Jul 12 17:42:25.342110 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.341980 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:25.342363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.342327 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.342363 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.342355 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.342573 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.342370 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.342573 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.342382 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c3c64c-ca69-4527-ac43-76ebfc46a520-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:42:25.342696 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.342658 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-data" (OuterVolumeSpecName: "istio-data") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:25.343819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.343789 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:25.343819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.343803 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:42:25.343819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.343815 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-token" (OuterVolumeSpecName: "istio-token") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:42:25.344021 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.343872 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-kube-api-access-t7r9b" (OuterVolumeSpecName: "kube-api-access-t7r9b") pod "48c3c64c-ca69-4527-ac43-76ebfc46a520" (UID: "48c3c64c-ca69-4527-ac43-76ebfc46a520"). InnerVolumeSpecName "kube-api-access-t7r9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:42:25.443245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.443197 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c3c64c-ca69-4527-ac43-76ebfc46a520-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.443245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.443235 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.443245 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.443246 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7r9b\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-kube-api-access-t7r9b\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.443481 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.443255 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.443481 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.443264 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.443481 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.443272 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c3c64c-ca69-4527-ac43-76ebfc46a520-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:25.763973 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.763939 2573 generic.go:356] "Generic (PLEG): container finished" podID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerID="989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c" exitCode=0 Jul 12 17:42:25.764173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.763998 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" Jul 12 17:42:25.764173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.764016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" event={"ID":"48c3c64c-ca69-4527-ac43-76ebfc46a520","Type":"ContainerDied","Data":"989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c"} Jul 12 17:42:25.764173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.764072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g" event={"ID":"48c3c64c-ca69-4527-ac43-76ebfc46a520","Type":"ContainerDied","Data":"46cd110947e2fef944ede29592a4ed5622373ca61ee02a87b5437aa3d3c5b823"} Jul 12 17:42:25.764173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.764102 2573 scope.go:117] "RemoveContainer" containerID="989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c" Jul 12 17:42:25.773421 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.773399 2573 scope.go:117] "RemoveContainer" containerID="32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d" Jul 12 17:42:25.781949 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.781931 2573 scope.go:117] "RemoveContainer" containerID="989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c" Jul 12 17:42:25.782197 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:42:25.782177 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c\": container with ID starting with 989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c not found: ID does not exist" containerID="989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c" Jul 12 17:42:25.782270 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.782210 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c"} err="failed to get container status \"989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c\": rpc error: code = NotFound desc = could not find container \"989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c\": container with ID starting with 989bffddde8bd179e46eec4f32baa2fa9377347742ffe7975b5b2ce856f3122c not found: ID does not exist" Jul 12 17:42:25.782270 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.782233 2573 scope.go:117] "RemoveContainer" containerID="32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d" Jul 12 17:42:25.782524 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:42:25.782507 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d\": container with ID starting with 32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d not found: ID does not exist" containerID="32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d" Jul 12 17:42:25.782565 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.782531 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d"} err="failed to get container status \"32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d\": rpc error: code = NotFound desc = could not find container \"32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d\": container with ID starting with 32debaeb1bfe6beec965054f2ac2ba5cc7ac9a77d9628f3d50d9e5bdeda3385d not found: ID does not exist" Jul 12 17:42:25.786441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.786385 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g"] Jul 12 17:42:25.789260 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:25.789237 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-444d10-openshift-default-7d7ddc4877-tvq6g"] Jul 12 17:42:27.039430 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:27.039393 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" path="/var/lib/kubelet/pods/48c3c64c-ca69-4527-ac43-76ebfc46a520/volumes" Jul 12 17:42:30.693689 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:30.693653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:35.402431 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:35.402388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:41.084887 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:41.084848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:41.141432 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:41.141391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:46.139702 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:46.139662 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj"] Jul 12 17:42:46.140206 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:46.139989 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" containerID="cri-o://d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689" gracePeriod=30 Jul 12 17:42:46.165130 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:46.165098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:46.412215 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:46.412181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:46.801529 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:46.801436 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:47.130171 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.130088 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz"] Jul 12 17:42:47.130875 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.130850 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" Jul 12 17:42:47.131113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131018 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" Jul 12 17:42:47.131113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131069 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a70c23e-b388-4667-8b94-ab4bf69ff144" containerName="revoke-keys" Jul 12 17:42:47.131113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131078 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a70c23e-b388-4667-8b94-ab4bf69ff144" containerName="revoke-keys" Jul 12 17:42:47.131113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131107 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" Jul 12 17:42:47.131113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131116 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" Jul 12 17:42:47.131406 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131256 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" Jul 12 17:42:47.131406 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.131272 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a70c23e-b388-4667-8b94-ab4bf69ff144" containerName="revoke-keys" Jul 12 17:42:47.136143 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.136120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.138611 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.138589 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e767e8-openshift-default-dockercfg-9dw2l\"" Jul 12 17:42:47.144771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.144749 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz"] Jul 12 17:42:47.162749 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.162695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:47.252319 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-credential-socket\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252319 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfb805a-d0ef-4a57-87a4-04077636f13a-istiod-ca-cert\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252541 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-envoy\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252541 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-data\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252632 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-certs\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252632 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-podinfo\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252632 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-token\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-socket\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.252742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.252710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65kv\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-kube-api-access-s65kv\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.353996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-credential-socket\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354043 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfb805a-d0ef-4a57-87a4-04077636f13a-istiod-ca-cert\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-envoy\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-data\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-certs\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-podinfo\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-token\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-socket\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354470 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65kv\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-kube-api-access-s65kv\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354674 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-credential-socket\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354755 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-data\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.354941 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.354891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-certs\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.355038 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.355009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfb805a-d0ef-4a57-87a4-04077636f13a-istiod-ca-cert\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.355166 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.355055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-socket\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.356577 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.356558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-envoy\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.357079 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.357058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-podinfo\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.362232 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.362205 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65kv\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-kube-api-access-s65kv\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.362457 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.362439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-token\") pod \"e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.448061 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.448024 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:47.542703 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.541420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:47.597344 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.597320 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz"] Jul 12 17:42:47.599844 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:42:47.599791 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfb805a_d0ef_4a57_87a4_04077636f13a.slice/crio-a5b1a59e7c7b41fa74d73f1c76adb5a91cfb7d7b9fb5f9e96220335a60362f1d WatchSource:0}: Error finding container a5b1a59e7c7b41fa74d73f1c76adb5a91cfb7d7b9fb5f9e96220335a60362f1d: Status 404 returned error can't find the container with id a5b1a59e7c7b41fa74d73f1c76adb5a91cfb7d7b9fb5f9e96220335a60362f1d Jul 12 17:42:47.602040 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.602008 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:42:47.602145 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.602096 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:42:47.602212 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.602141 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:42:47.890147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.890045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" event={"ID":"3bfb805a-d0ef-4a57-87a4-04077636f13a","Type":"ContainerStarted","Data":"efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108"} Jul 12 17:42:47.890147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.890106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" event={"ID":"3bfb805a-d0ef-4a57-87a4-04077636f13a","Type":"ContainerStarted","Data":"a5b1a59e7c7b41fa74d73f1c76adb5a91cfb7d7b9fb5f9e96220335a60362f1d"} Jul 12 17:42:47.910151 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:47.910100 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podStartSLOduration=0.910084391 podStartE2EDuration="910.084391ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:42:47.908494564 +0000 UTC m=+2375.403401006" watchObservedRunningTime="2026-07-12 17:42:47.910084391 +0000 UTC m=+2375.404990846" Jul 12 17:42:48.448388 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:48.448349 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:48.453741 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:48.453712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:48.896733 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:48.896631 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:48.897996 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:48.897977 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:42:49.901019 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:49.900985 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:49.988561 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:49.988529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:50.059562 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:50.059522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:50.262291 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:50.262256 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:50.554362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:50.554267 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:50.658537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:50.658502 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:51.408326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.408298 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:42:51.498789 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.498753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-podinfo\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.498805 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdrcx\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-kube-api-access-pdrcx\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.498844 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-token\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.498870 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istiod-ca-cert\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.498900 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-envoy\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.498973 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-credential-socket\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499021 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-certs\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499045 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-socket\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499266 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499087 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-data\") pod \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\" (UID: \"d5dd8eb0-6b57-4be3-b11d-ca519163a995\") " Jul 12 17:42:51.499427 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499297 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:51.499427 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499409 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:51.499546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499494 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.499546 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499514 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.499643 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499560 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:51.499643 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.499612 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:42:51.500289 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.500261 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:51.501094 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.501068 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:42:51.501172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.501136 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-kube-api-access-pdrcx" (OuterVolumeSpecName: "kube-api-access-pdrcx") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "kube-api-access-pdrcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:42:51.501172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.501156 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:42:51.501330 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.501316 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5dd8eb0-6b57-4be3-b11d-ca519163a995" (UID: "d5dd8eb0-6b57-4be3-b11d-ca519163a995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:42:51.601030 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.600995 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.601030 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.601024 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdrcx\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-kube-api-access-pdrcx\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.601030 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.601033 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.601274 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.601043 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.601274 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.601051 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.601274 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.601059 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.601274 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.601067 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8eb0-6b57-4be3-b11d-ca519163a995-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:42:51.917506 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.917473 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerID="d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689" exitCode=0 Jul 12 17:42:51.917694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.917528 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" Jul 12 17:42:51.917694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.917556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" event={"ID":"d5dd8eb0-6b57-4be3-b11d-ca519163a995","Type":"ContainerDied","Data":"d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689"} Jul 12 17:42:51.917694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.917609 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj" event={"ID":"d5dd8eb0-6b57-4be3-b11d-ca519163a995","Type":"ContainerDied","Data":"ede275110b3f827a6b15d96502f6ff62c3185bcc1041575ae0ab24793ea02f41"} Jul 12 17:42:51.917694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.917633 2573 scope.go:117] "RemoveContainer" containerID="d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689" Jul 12 17:42:51.927621 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.927601 2573 scope.go:117] "RemoveContainer" containerID="cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914" Jul 12 17:42:51.937276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.937257 2573 scope.go:117] "RemoveContainer" containerID="d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689" Jul 12 17:42:51.937610 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:42:51.937586 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689\": container with ID starting with d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689 not found: ID does not exist" containerID="d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689" Jul 12 17:42:51.937668 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.937627 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689"} err="failed to get container status \"d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689\": rpc error: code = NotFound desc = could not find container \"d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689\": container with ID starting with d779c7f5a0d907f9a224a4d3041a5821218c5a483f396a45317964896df6b689 not found: ID does not exist" Jul 12 17:42:51.937668 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.937652 2573 scope.go:117] "RemoveContainer" containerID="cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914" Jul 12 17:42:51.938037 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:42:51.937967 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914\": container with ID starting with cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914 not found: ID does not exist" containerID="cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914" Jul 12 17:42:51.938037 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.938005 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914"} err="failed to get container status \"cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914\": rpc error: code = NotFound desc = could not find container \"cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914\": container with ID starting with cee87aa58fd5cb9a44b29f1f805db53c16341d057809c31019fd1e9fedd48914 not found: ID does not exist" Jul 12 17:42:51.939973 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.939950 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj"] Jul 12 17:42:51.941995 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:51.941977 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2484ca-openshift-default-5ccd87ddf7-tw2kj"] Jul 12 17:42:53.040656 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:53.040623 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" path="/var/lib/kubelet/pods/d5dd8eb0-6b57-4be3-b11d-ca519163a995/volumes" Jul 12 17:42:54.480858 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:54.480825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:54.845261 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:54.845166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:55.143988 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.143830 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972"] Jul 12 17:42:55.144528 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.144511 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" Jul 12 17:42:55.144575 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.144532 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" Jul 12 17:42:55.144676 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.144665 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c64c-ca69-4527-ac43-76ebfc46a520" containerName="istio-proxy" Jul 12 17:42:55.144711 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.144681 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" Jul 12 17:42:55.149900 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.149873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.152423 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.152397 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-9c75b6-openshift-default-dockercfg-hnhh7\"" Jul 12 17:42:55.164560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.164530 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972"] Jul 12 17:42:55.167870 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.167844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:55.236468 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-socket\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.236680 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-podinfo\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.236680 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-certs\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.236802 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istiod-ca-cert\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.236802 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gn54\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-kube-api-access-7gn54\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.236905 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-envoy\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.236905 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-token\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.237029 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-data\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.237029 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.236995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-credential-socket\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.268559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.268520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:55.338308 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-data\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-credential-socket\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-socket\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-podinfo\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-certs\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338513 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istiod-ca-cert\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gn54\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-kube-api-access-7gn54\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-envoy\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338784 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-token\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338986 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338878 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-data\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.338986 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.338966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-certs\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.339383 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.339351 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-credential-socket\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.339679 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.339625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istiod-ca-cert\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.339955 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.339877 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-socket\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.342077 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.342041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-envoy\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.342548 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.342525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-podinfo\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.346723 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.346689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-token\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.346846 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.346821 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gn54\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-kube-api-access-7gn54\") pod \"e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.462097 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.462057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:55.607527 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.607504 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972"] Jul 12 17:42:55.609701 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:42:55.609673 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5bf9bb_0276_46c9_a85c_c96cf8f18b71.slice/crio-60e9c160c9166a01ad1f415a6669d5d28091fb80a08e9f0abdb96878770d99f0 WatchSource:0}: Error finding container 60e9c160c9166a01ad1f415a6669d5d28091fb80a08e9f0abdb96878770d99f0: Status 404 returned error can't find the container with id 60e9c160c9166a01ad1f415a6669d5d28091fb80a08e9f0abdb96878770d99f0 Jul 12 17:42:55.612358 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.612326 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:42:55.612454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.612401 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:42:55.612454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.612429 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:42:55.942685 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.942651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" event={"ID":"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71","Type":"ContainerStarted","Data":"96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee"} Jul 12 17:42:55.942906 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.942715 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" event={"ID":"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71","Type":"ContainerStarted","Data":"60e9c160c9166a01ad1f415a6669d5d28091fb80a08e9f0abdb96878770d99f0"} Jul 12 17:42:55.962707 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:55.962660 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" podStartSLOduration=0.962642889 podStartE2EDuration="962.642889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:42:55.961156581 +0000 UTC m=+2383.456063044" watchObservedRunningTime="2026-07-12 17:42:55.962642889 +0000 UTC m=+2383.457549327" Jul 12 17:42:56.462884 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:56.462845 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:56.468224 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:56.468198 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:56.956141 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:56.956102 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:56.958690 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:56.958661 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:42:57.896773 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:57.896735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:57.990474 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:57.990439 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:58.060653 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:58.060623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:58.209149 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:58.209115 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:58.533200 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:58.533106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:42:58.643926 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:42:58.643878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:02.444528 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:02.444485 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:03.898058 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:03.897963 2573 patch_prober.go:28] interesting pod/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 12 17:43:03.898058 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:03.898034 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 12 17:43:03.999503 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:03.999470 2573 generic.go:356] "Generic (PLEG): container finished" podID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerID="efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108" exitCode=0 Jul 12 17:43:03.999679 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:03.999545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" event={"ID":"3bfb805a-d0ef-4a57-87a4-04077636f13a","Type":"ContainerDied","Data":"efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108"} Jul 12 17:43:03.999970 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:03.999955 2573 scope.go:117] "RemoveContainer" containerID="efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108" Jul 12 17:43:04.001004 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:04.000981 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:43:04.001074 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:04.001042 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:43:04.001129 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:04.001074 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:43:05.007050 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:05.007015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" event={"ID":"3bfb805a-d0ef-4a57-87a4-04077636f13a","Type":"ContainerStarted","Data":"0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3"} Jul 12 17:43:05.449110 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:05.449068 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:43:05.449326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:05.449220 2573 patch_prober.go:28] interesting pod/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 12 17:43:05.449326 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:05.449307 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 12 17:43:06.448893 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:06.448847 2573 patch_prober.go:28] interesting pod/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 12 17:43:06.449403 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:06.448951 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 12 17:43:07.449318 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:07.449272 2573 patch_prober.go:28] interesting pod/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 12 17:43:07.449991 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:07.449957 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 12 17:43:08.449390 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:08.449347 2573 patch_prober.go:28] interesting pod/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 12 17:43:08.449877 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:08.449433 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 12 17:43:09.452783 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:09.452750 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:43:09.453247 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:09.453226 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:43:09.453905 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:09.453884 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:43:11.044005 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.043973 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerID="96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee" exitCode=0 Jul 12 17:43:11.044491 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.044045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" event={"ID":"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71","Type":"ContainerDied","Data":"96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee"} Jul 12 17:43:11.044561 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.044510 2573 scope.go:117] "RemoveContainer" containerID="96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee" Jul 12 17:43:11.045495 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.045473 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:43:11.045557 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.045538 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:43:11.045596 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.045568 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 12 17:43:11.244801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.244676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:11.347872 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.347823 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:11.956886 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:11.956827 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:43:12.055397 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:12.055352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" event={"ID":"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71","Type":"ContainerStarted","Data":"7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21"} Jul 12 17:43:12.463008 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:12.462875 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:43:12.464568 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:12.464529 2573 patch_prober.go:28] interesting pod/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 12 17:43:12.464750 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:12.464628 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 12 17:43:13.463316 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:13.463258 2573 patch_prober.go:28] interesting pod/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 12 17:43:13.463743 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:13.463344 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 12 17:43:13.615870 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:13.615786 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:43:13.627461 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:13.627436 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:43:14.463125 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:14.463086 2573 patch_prober.go:28] interesting pod/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 12 17:43:14.463418 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:14.463392 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 12 17:43:15.463306 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:15.463266 2573 patch_prober.go:28] interesting pod/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 12 17:43:15.463492 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:15.463326 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 12 17:43:16.467147 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:16.467118 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:43:16.467563 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:16.467355 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:43:16.467995 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:16.467979 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:43:19.395155 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:19.395046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:19.495462 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:19.495422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:22.519328 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:22.519287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:28.548643 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:28.548608 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:43:56.235500 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:43:56.235377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:44:44.984157 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:44:44.984115 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:44:45.037053 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:44:45.037007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:45:13.354990 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:45:13.354894 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:45:13.455794 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:45:13.455754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:45:53.589634 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:45:53.589601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:45:54.588300 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:45:54.588263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:46:08.918302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:46:08.918262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:46:13.076250 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:46:13.076216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:46:25.327945 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:46:25.327890 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:46:59.011653 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:46:59.011534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:47:52.738787 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:47:52.738725 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:47:54.177512 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:47:54.177450 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:47:56.893716 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:47:56.893681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:48:11.928338 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:11.928301 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:48:12.029973 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:12.029939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:48:13.651802 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:13.651771 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:48:13.670352 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:13.670323 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvmx2_fb942b28-ce95-4401-bb85-b25474302d61/ovn-acl-logging/0.log" Jul 12 17:48:20.105757 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:20.105719 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:48:20.205939 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:20.205873 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:48:27.558522 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:27.558485 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:48:39.791125 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:48:39.791090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:49:35.099808 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:35.099720 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:49:35.990960 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:35.990927 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:49:36.996001 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:36.995958 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7477-140060956412336"] Jul 12 17:49:36.996608 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:36.996586 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" Jul 12 17:49:36.996703 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:36.996611 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" Jul 12 17:49:36.996772 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:36.996758 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5dd8eb0-6b57-4be3-b11d-ca519163a995" containerName="istio-proxy" Jul 12 17:49:37.000015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:36.999992 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:37.002891 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.002868 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-vv9ps\"" Jul 12 17:49:37.004829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.004803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060956412336"] Jul 12 17:49:37.123397 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.123266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnn97\" (UniqueName: \"kubernetes.io/projected/80ea2699-b427-4fd7-90ea-11979c22a89d-kube-api-access-bnn97\") pod \"test-curl-7477-140060956412336\" (UID: \"80ea2699-b427-4fd7-90ea-11979c22a89d\") " pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:37.224496 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.224443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnn97\" (UniqueName: \"kubernetes.io/projected/80ea2699-b427-4fd7-90ea-11979c22a89d-kube-api-access-bnn97\") pod \"test-curl-7477-140060956412336\" (UID: \"80ea2699-b427-4fd7-90ea-11979c22a89d\") " pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:37.232837 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.232797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnn97\" (UniqueName: \"kubernetes.io/projected/80ea2699-b427-4fd7-90ea-11979c22a89d-kube-api-access-bnn97\") pod \"test-curl-7477-140060956412336\" (UID: \"80ea2699-b427-4fd7-90ea-11979c22a89d\") " pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:37.311645 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.311563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:37.451669 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.451637 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060956412336"] Jul 12 17:49:37.453304 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:49:37.453278 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80ea2699_b427_4fd7_90ea_11979c22a89d.slice/crio-42786572b37c42856b656d7319a8ed361b37778d8ce92f3c58e1e511e8889b43 WatchSource:0}: Error finding container 42786572b37c42856b656d7319a8ed361b37778d8ce92f3c58e1e511e8889b43: Status 404 returned error can't find the container with id 42786572b37c42856b656d7319a8ed361b37778d8ce92f3c58e1e511e8889b43 Jul 12 17:49:37.454607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:37.454587 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 17:49:38.243173 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:38.243136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060956412336" event={"ID":"80ea2699-b427-4fd7-90ea-11979c22a89d","Type":"ContainerStarted","Data":"649769c38ff2466a65f6c3cce8c7c3dd7b9e021e38409e3b616fbbddb14448b0"} Jul 12 17:49:38.243655 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:38.243184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060956412336" event={"ID":"80ea2699-b427-4fd7-90ea-11979c22a89d","Type":"ContainerStarted","Data":"42786572b37c42856b656d7319a8ed361b37778d8ce92f3c58e1e511e8889b43"} Jul 12 17:49:38.257774 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:38.257714 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/test-curl-7477-140060956412336" podStartSLOduration=1.550450499 podStartE2EDuration="2.25769237s" totalImagesPullingTime="707.241871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:49:38.257020021 +0000 UTC m=+2785.751926498" watchObservedRunningTime="2026-07-12 17:49:38.25769237 +0000 UTC m=+2785.752598807" Jul 12 17:49:38.351536 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:38.351509 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7477-140060956412336_80ea2699-b427-4fd7-90ea-11979c22a89d/test-curl-7477-140060956412336/0.log" Jul 12 17:49:39.250763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:39.250703 2573 generic.go:356] "Generic (PLEG): container finished" podID="80ea2699-b427-4fd7-90ea-11979c22a89d" containerID="649769c38ff2466a65f6c3cce8c7c3dd7b9e021e38409e3b616fbbddb14448b0" exitCode=0 Jul 12 17:49:39.250763 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:39.250748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060956412336" event={"ID":"80ea2699-b427-4fd7-90ea-11979c22a89d","Type":"ContainerDied","Data":"649769c38ff2466a65f6c3cce8c7c3dd7b9e021e38409e3b616fbbddb14448b0"} Jul 12 17:49:40.411716 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.411693 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:40.442880 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.442846 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7477-140060956412336"] Jul 12 17:49:40.446991 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.446964 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7477-140060956412336"] Jul 12 17:49:40.455064 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.455028 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnn97\" (UniqueName: \"kubernetes.io/projected/80ea2699-b427-4fd7-90ea-11979c22a89d-kube-api-access-bnn97\") pod \"80ea2699-b427-4fd7-90ea-11979c22a89d\" (UID: \"80ea2699-b427-4fd7-90ea-11979c22a89d\") " Jul 12 17:49:40.457076 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.457049 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ea2699-b427-4fd7-90ea-11979c22a89d-kube-api-access-bnn97" (OuterVolumeSpecName: "kube-api-access-bnn97") pod "80ea2699-b427-4fd7-90ea-11979c22a89d" (UID: "80ea2699-b427-4fd7-90ea-11979c22a89d"). InnerVolumeSpecName "kube-api-access-bnn97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:49:40.555832 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.555737 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnn97\" (UniqueName: \"kubernetes.io/projected/80ea2699-b427-4fd7-90ea-11979c22a89d-kube-api-access-bnn97\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:49:40.564397 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.564365 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7477-140060956398512"] Jul 12 17:49:40.564873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.564859 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ea2699-b427-4fd7-90ea-11979c22a89d" containerName="test-curl-7477-140060956412336" Jul 12 17:49:40.564946 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.564875 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ea2699-b427-4fd7-90ea-11979c22a89d" containerName="test-curl-7477-140060956412336" Jul 12 17:49:40.564992 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.564981 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ea2699-b427-4fd7-90ea-11979c22a89d" containerName="test-curl-7477-140060956412336" Jul 12 17:49:40.576670 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.576631 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060956398512"] Jul 12 17:49:40.576847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.576758 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:40.656449 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.656413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ht4\" (UniqueName: \"kubernetes.io/projected/505338b1-e18b-4fed-b724-58cf7f5b55c5-kube-api-access-x9ht4\") pod \"test-curl-7477-140060956398512\" (UID: \"505338b1-e18b-4fed-b724-58cf7f5b55c5\") " pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:40.757971 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.757938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ht4\" (UniqueName: \"kubernetes.io/projected/505338b1-e18b-4fed-b724-58cf7f5b55c5-kube-api-access-x9ht4\") pod \"test-curl-7477-140060956398512\" (UID: \"505338b1-e18b-4fed-b724-58cf7f5b55c5\") " pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:40.766114 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.766088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ht4\" (UniqueName: \"kubernetes.io/projected/505338b1-e18b-4fed-b724-58cf7f5b55c5-kube-api-access-x9ht4\") pod \"test-curl-7477-140060956398512\" (UID: \"505338b1-e18b-4fed-b724-58cf7f5b55c5\") " pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:40.809786 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.809695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:49:40.890295 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:40.890260 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:41.027889 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:41.027861 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060956398512"] Jul 12 17:49:41.029382 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:49:41.029355 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod505338b1_e18b_4fed_b724_58cf7f5b55c5.slice/crio-a6fd78236d5fa8fdfbf0b584c3af9ab5117f6b78c9a1b676ab775cb612d63403 WatchSource:0}: Error finding container a6fd78236d5fa8fdfbf0b584c3af9ab5117f6b78c9a1b676ab775cb612d63403: Status 404 returned error can't find the container with id a6fd78236d5fa8fdfbf0b584c3af9ab5117f6b78c9a1b676ab775cb612d63403 Jul 12 17:49:41.041340 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:41.041303 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ea2699-b427-4fd7-90ea-11979c22a89d" path="/var/lib/kubelet/pods/80ea2699-b427-4fd7-90ea-11979c22a89d/volumes" Jul 12 17:49:41.264997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:41.264958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060956398512" event={"ID":"505338b1-e18b-4fed-b724-58cf7f5b55c5","Type":"ContainerStarted","Data":"a6fd78236d5fa8fdfbf0b584c3af9ab5117f6b78c9a1b676ab775cb612d63403"} Jul 12 17:49:41.266409 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:41.266384 2573 scope.go:117] "RemoveContainer" containerID="649769c38ff2466a65f6c3cce8c7c3dd7b9e021e38409e3b616fbbddb14448b0" Jul 12 17:49:41.266550 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:41.266388 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956412336" Jul 12 17:49:42.274938 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:42.274881 2573 generic.go:356] "Generic (PLEG): container finished" podID="505338b1-e18b-4fed-b724-58cf7f5b55c5" containerID="e69c846dea3e2051a6945556a4ac1a1fe027be64730a4e2cbc1c1647a02a146d" exitCode=0 Jul 12 17:49:42.275346 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:42.274965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060956398512" event={"ID":"505338b1-e18b-4fed-b724-58cf7f5b55c5","Type":"ContainerDied","Data":"e69c846dea3e2051a6945556a4ac1a1fe027be64730a4e2cbc1c1647a02a146d"} Jul 12 17:49:43.432386 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.432356 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:43.450809 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.450775 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7477-140060956398512_505338b1-e18b-4fed-b724-58cf7f5b55c5/test-curl-7477-140060956398512/0.log" Jul 12 17:49:43.471894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.471863 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7477-140060956398512"] Jul 12 17:49:43.476044 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.476016 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7477-140060956398512"] Jul 12 17:49:43.485572 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.485553 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9ht4\" (UniqueName: \"kubernetes.io/projected/505338b1-e18b-4fed-b724-58cf7f5b55c5-kube-api-access-x9ht4\") pod \"505338b1-e18b-4fed-b724-58cf7f5b55c5\" (UID: \"505338b1-e18b-4fed-b724-58cf7f5b55c5\") " Jul 12 17:49:43.487635 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.487613 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505338b1-e18b-4fed-b724-58cf7f5b55c5-kube-api-access-x9ht4" (OuterVolumeSpecName: "kube-api-access-x9ht4") pod "505338b1-e18b-4fed-b724-58cf7f5b55c5" (UID: "505338b1-e18b-4fed-b724-58cf7f5b55c5"). InnerVolumeSpecName "kube-api-access-x9ht4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:49:43.586433 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.586347 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ht4\" (UniqueName: \"kubernetes.io/projected/505338b1-e18b-4fed-b724-58cf7f5b55c5-kube-api-access-x9ht4\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:49:43.632777 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.632739 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:43.633226 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.633211 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505338b1-e18b-4fed-b724-58cf7f5b55c5" containerName="test-curl-7477-140060956398512" Jul 12 17:49:43.633277 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.633227 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="505338b1-e18b-4fed-b724-58cf7f5b55c5" containerName="test-curl-7477-140060956398512" Jul 12 17:49:43.633324 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.633315 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="505338b1-e18b-4fed-b724-58cf7f5b55c5" containerName="test-curl-7477-140060956398512" Jul 12 17:49:43.636306 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.636283 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:43.640265 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.640247 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:43.687372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.687343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6stj\" (UniqueName: \"kubernetes.io/projected/a6658d43-6267-4a99-9046-9f7d7dcb2402-kube-api-access-m6stj\") pod \"test-curl-7477-140060958484928\" (UID: \"a6658d43-6267-4a99-9046-9f7d7dcb2402\") " pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:43.788823 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.788786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6stj\" (UniqueName: \"kubernetes.io/projected/a6658d43-6267-4a99-9046-9f7d7dcb2402-kube-api-access-m6stj\") pod \"test-curl-7477-140060958484928\" (UID: \"a6658d43-6267-4a99-9046-9f7d7dcb2402\") " pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:43.797189 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.797167 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6stj\" (UniqueName: \"kubernetes.io/projected/a6658d43-6267-4a99-9046-9f7d7dcb2402-kube-api-access-m6stj\") pod \"test-curl-7477-140060958484928\" (UID: \"a6658d43-6267-4a99-9046-9f7d7dcb2402\") " pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:43.948146 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:43.948117 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:44.085871 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:44.085844 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:44.087187 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:49:44.087166 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6658d43_6267_4a99_9046_9f7d7dcb2402.slice/crio-a60d9cb922b6147804316523fb40bb903d6b34f5abc18a1762fe1b6c9ef180a9 WatchSource:0}: Error finding container a60d9cb922b6147804316523fb40bb903d6b34f5abc18a1762fe1b6c9ef180a9: Status 404 returned error can't find the container with id a60d9cb922b6147804316523fb40bb903d6b34f5abc18a1762fe1b6c9ef180a9 Jul 12 17:49:44.288636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:44.288551 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:44.288636 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:44.288600 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6fd78236d5fa8fdfbf0b584c3af9ab5117f6b78c9a1b676ab775cb612d63403" Jul 12 17:49:44.289873 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:44.289848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484928" event={"ID":"a6658d43-6267-4a99-9046-9f7d7dcb2402","Type":"ContainerStarted","Data":"a60d9cb922b6147804316523fb40bb903d6b34f5abc18a1762fe1b6c9ef180a9"} Jul 12 17:49:44.299464 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:49:44.299437 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7477-140060956398512\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="505338b1-e18b-4fed-b724-58cf7f5b55c5" pod="openshift-ingress/test-curl-7477-140060956398512" Jul 12 17:49:45.040595 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:45.040556 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505338b1-e18b-4fed-b724-58cf7f5b55c5" path="/var/lib/kubelet/pods/505338b1-e18b-4fed-b724-58cf7f5b55c5/volumes" Jul 12 17:49:45.297302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:45.297205 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6658d43-6267-4a99-9046-9f7d7dcb2402" containerID="ed606fa803786fb5013765f4120b160ea066931c9640c9f527f75a69cfab18d2" exitCode=0 Jul 12 17:49:45.297302 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:45.297247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484928" event={"ID":"a6658d43-6267-4a99-9046-9f7d7dcb2402","Type":"ContainerDied","Data":"ed606fa803786fb5013765f4120b160ea066931c9640c9f527f75a69cfab18d2"} Jul 12 17:49:46.449438 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.449410 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:46.467372 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.467342 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7477-140060958484928_a6658d43-6267-4a99-9046-9f7d7dcb2402/test-curl-7477-140060958484928/0.log" Jul 12 17:49:46.497816 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.497786 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:46.502121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.502088 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:46.513845 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.513819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6stj\" (UniqueName: \"kubernetes.io/projected/a6658d43-6267-4a99-9046-9f7d7dcb2402-kube-api-access-m6stj\") pod \"a6658d43-6267-4a99-9046-9f7d7dcb2402\" (UID: \"a6658d43-6267-4a99-9046-9f7d7dcb2402\") " Jul 12 17:49:46.515878 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.515851 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6658d43-6267-4a99-9046-9f7d7dcb2402-kube-api-access-m6stj" (OuterVolumeSpecName: "kube-api-access-m6stj") pod "a6658d43-6267-4a99-9046-9f7d7dcb2402" (UID: "a6658d43-6267-4a99-9046-9f7d7dcb2402"). InnerVolumeSpecName "kube-api-access-m6stj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:49:46.615463 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.615369 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6stj\" (UniqueName: \"kubernetes.io/projected/a6658d43-6267-4a99-9046-9f7d7dcb2402-kube-api-access-m6stj\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:49:46.617835 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.617804 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:46.618360 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.618341 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6658d43-6267-4a99-9046-9f7d7dcb2402" containerName="test-curl-7477-140060958484928" Jul 12 17:49:46.618439 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.618361 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6658d43-6267-4a99-9046-9f7d7dcb2402" containerName="test-curl-7477-140060958484928" Jul 12 17:49:46.618498 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.618480 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6658d43-6267-4a99-9046-9f7d7dcb2402" containerName="test-curl-7477-140060958484928" Jul 12 17:49:46.621742 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.621719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:46.625845 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.625821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:46.716755 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.716724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kzq\" (UniqueName: \"kubernetes.io/projected/befa9c15-faf6-4936-936d-dad49633a997-kube-api-access-86kzq\") pod \"test-curl-7477-140060958484928\" (UID: \"befa9c15-faf6-4936-936d-dad49633a997\") " pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:46.818123 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.818086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kzq\" (UniqueName: \"kubernetes.io/projected/befa9c15-faf6-4936-936d-dad49633a997-kube-api-access-86kzq\") pod \"test-curl-7477-140060958484928\" (UID: \"befa9c15-faf6-4936-936d-dad49633a997\") " pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:46.825875 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.825845 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kzq\" (UniqueName: \"kubernetes.io/projected/befa9c15-faf6-4936-936d-dad49633a997-kube-api-access-86kzq\") pod \"test-curl-7477-140060958484928\" (UID: \"befa9c15-faf6-4936-936d-dad49633a997\") " pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:46.933353 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:46.933318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:47.043490 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:47.043448 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6658d43-6267-4a99-9046-9f7d7dcb2402" path="/var/lib/kubelet/pods/a6658d43-6267-4a99-9046-9f7d7dcb2402/volumes" Jul 12 17:49:47.070401 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:47.070372 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:47.071420 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:49:47.071379 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbefa9c15_faf6_4936_936d_dad49633a997.slice/crio-f93606f50d0881f5f33c46ce736be7a694cb8e833d1ef69d2e69c43b1b2bbfbd WatchSource:0}: Error finding container f93606f50d0881f5f33c46ce736be7a694cb8e833d1ef69d2e69c43b1b2bbfbd: Status 404 returned error can't find the container with id f93606f50d0881f5f33c46ce736be7a694cb8e833d1ef69d2e69c43b1b2bbfbd Jul 12 17:49:47.309779 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:47.309703 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:47.309779 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:47.309740 2573 scope.go:117] "RemoveContainer" containerID="ed606fa803786fb5013765f4120b160ea066931c9640c9f527f75a69cfab18d2" Jul 12 17:49:47.310997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:47.310932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484928" event={"ID":"befa9c15-faf6-4936-936d-dad49633a997","Type":"ContainerStarted","Data":"f93606f50d0881f5f33c46ce736be7a694cb8e833d1ef69d2e69c43b1b2bbfbd"} Jul 12 17:49:48.319113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:48.319081 2573 generic.go:356] "Generic (PLEG): container finished" podID="befa9c15-faf6-4936-936d-dad49633a997" containerID="5633fede23b20223d1ce6d086f878ccb5311ed0a73a164692987275b91591dee" exitCode=0 Jul 12 17:49:48.319501 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:48.319124 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484928" event={"ID":"befa9c15-faf6-4936-936d-dad49633a997","Type":"ContainerDied","Data":"5633fede23b20223d1ce6d086f878ccb5311ed0a73a164692987275b91591dee"} Jul 12 17:49:49.480276 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.480246 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:49.500819 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.500784 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7477-140060958484928_befa9c15-faf6-4936-936d-dad49633a997/test-curl-7477-140060958484928/0.log" Jul 12 17:49:49.522252 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.522217 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:49.526699 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.526670 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484928"] Jul 12 17:49:49.544483 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.544453 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86kzq\" (UniqueName: \"kubernetes.io/projected/befa9c15-faf6-4936-936d-dad49633a997-kube-api-access-86kzq\") pod \"befa9c15-faf6-4936-936d-dad49633a997\" (UID: \"befa9c15-faf6-4936-936d-dad49633a997\") " Jul 12 17:49:49.546549 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.546520 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befa9c15-faf6-4936-936d-dad49633a997-kube-api-access-86kzq" (OuterVolumeSpecName: "kube-api-access-86kzq") pod "befa9c15-faf6-4936-936d-dad49633a997" (UID: "befa9c15-faf6-4936-936d-dad49633a997"). InnerVolumeSpecName "kube-api-access-86kzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:49:49.646186 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.646101 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86kzq\" (UniqueName: \"kubernetes.io/projected/befa9c15-faf6-4936-936d-dad49633a997-kube-api-access-86kzq\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:49:49.648006 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.647982 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"] Jul 12 17:49:49.648439 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.648427 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befa9c15-faf6-4936-936d-dad49633a997" containerName="test-curl-7477-140060958484928" Jul 12 17:49:49.648489 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.648441 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="befa9c15-faf6-4936-936d-dad49633a997" containerName="test-curl-7477-140060958484928" Jul 12 17:49:49.648525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.648503 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="befa9c15-faf6-4936-936d-dad49633a997" containerName="test-curl-7477-140060958484928" Jul 12 17:49:49.651570 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.651552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:49.655189 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.655164 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"] Jul 12 17:49:49.747133 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.747089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ldh\" (UniqueName: \"kubernetes.io/projected/f83212fd-b7db-46dd-9020-807e7d817dab-kube-api-access-q8ldh\") pod \"test-curl-7477-140060958484784\" (UID: \"f83212fd-b7db-46dd-9020-807e7d817dab\") " pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:49.848069 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.848032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ldh\" (UniqueName: \"kubernetes.io/projected/f83212fd-b7db-46dd-9020-807e7d817dab-kube-api-access-q8ldh\") pod \"test-curl-7477-140060958484784\" (UID: \"f83212fd-b7db-46dd-9020-807e7d817dab\") " pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:49.857771 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.857743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ldh\" (UniqueName: \"kubernetes.io/projected/f83212fd-b7db-46dd-9020-807e7d817dab-kube-api-access-q8ldh\") pod \"test-curl-7477-140060958484784\" (UID: \"f83212fd-b7db-46dd-9020-807e7d817dab\") " pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:49.963125 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:49.963090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:50.122078 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:50.122049 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"] Jul 12 17:49:50.123546 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:49:50.123518 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf83212fd_b7db_46dd_9020_807e7d817dab.slice/crio-813e4b61baea240e5170ff39686e2e281cf0dd6d251879b5e0bd00331603cdfc WatchSource:0}: Error finding container 813e4b61baea240e5170ff39686e2e281cf0dd6d251879b5e0bd00331603cdfc: Status 404 returned error can't find the container with id 813e4b61baea240e5170ff39686e2e281cf0dd6d251879b5e0bd00331603cdfc Jul 12 17:49:50.332182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:50.332088 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:50.332182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:50.332132 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93606f50d0881f5f33c46ce736be7a694cb8e833d1ef69d2e69c43b1b2bbfbd" Jul 12 17:49:50.333414 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:50.333376 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484784" event={"ID":"f83212fd-b7db-46dd-9020-807e7d817dab","Type":"ContainerStarted","Data":"813e4b61baea240e5170ff39686e2e281cf0dd6d251879b5e0bd00331603cdfc"} Jul 12 17:49:50.343612 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:49:50.343585 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7477-140060958484928\" is forbidden: User \"system:node:ip-10-0-139-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-89.ec2.internal' and this object" podUID="befa9c15-faf6-4936-936d-dad49633a997" pod="openshift-ingress/test-curl-7477-140060958484928" Jul 12 17:49:51.046012 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:51.045972 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="befa9c15-faf6-4936-936d-dad49633a997" path="/var/lib/kubelet/pods/befa9c15-faf6-4936-936d-dad49633a997/volumes" Jul 12 17:49:51.341055 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:51.340960 2573 generic.go:356] "Generic (PLEG): container finished" podID="f83212fd-b7db-46dd-9020-807e7d817dab" containerID="61cbf5d1463c091497504c1cf44666d597a818241c9f60f87088ef3600a6b60d" exitCode=0 Jul 12 17:49:51.341055 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:51.341046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484784" event={"ID":"f83212fd-b7db-46dd-9020-807e7d817dab","Type":"ContainerDied","Data":"61cbf5d1463c091497504c1cf44666d597a818241c9f60f87088ef3600a6b60d"} Jul 12 17:49:52.503151 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.503116 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:52.522454 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.522426 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7477-140060958484784_f83212fd-b7db-46dd-9020-807e7d817dab/test-curl-7477-140060958484784/0.log" Jul 12 17:49:52.547369 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.547329 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"] Jul 12 17:49:52.553198 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.553173 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"] Jul 12 17:49:52.578018 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.577991 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8ldh\" (UniqueName: \"kubernetes.io/projected/f83212fd-b7db-46dd-9020-807e7d817dab-kube-api-access-q8ldh\") pod \"f83212fd-b7db-46dd-9020-807e7d817dab\" (UID: \"f83212fd-b7db-46dd-9020-807e7d817dab\") " Jul 12 17:49:52.580034 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.580010 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83212fd-b7db-46dd-9020-807e7d817dab-kube-api-access-q8ldh" (OuterVolumeSpecName: "kube-api-access-q8ldh") pod "f83212fd-b7db-46dd-9020-807e7d817dab" (UID: "f83212fd-b7db-46dd-9020-807e7d817dab"). InnerVolumeSpecName "kube-api-access-q8ldh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:49:52.679714 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:52.679680 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ldh\" (UniqueName: \"kubernetes.io/projected/f83212fd-b7db-46dd-9020-807e7d817dab-kube-api-access-q8ldh\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:49:53.040941 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:53.040831 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83212fd-b7db-46dd-9020-807e7d817dab" path="/var/lib/kubelet/pods/f83212fd-b7db-46dd-9020-807e7d817dab/volumes" Jul 12 17:49:53.353341 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:53.353244 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784" Jul 12 17:49:53.353341 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:53.353282 2573 scope.go:117] "RemoveContainer" containerID="61cbf5d1463c091497504c1cf44666d597a818241c9f60f87088ef3600a6b60d" Jul 12 17:49:55.450154 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.450099 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz"] Jul 12 17:49:55.450728 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.450708 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83212fd-b7db-46dd-9020-807e7d817dab" containerName="test-curl-7477-140060958484784" Jul 12 17:49:55.450728 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.450730 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83212fd-b7db-46dd-9020-807e7d817dab" containerName="test-curl-7477-140060958484784" Jul 12 17:49:55.450844 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.450815 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83212fd-b7db-46dd-9020-807e7d817dab" containerName="test-curl-7477-140060958484784" Jul 12 17:49:55.455415 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.455394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:55.458091 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.458042 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5xzcz\"" Jul 12 17:49:55.460256 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.460218 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz"] Jul 12 17:49:55.606525 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.606482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd1531-3b66-4b65-b014-9ddb293af6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz\" (UID: \"7bcd1531-3b66-4b65-b014-9ddb293af6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:55.707543 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.707432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd1531-3b66-4b65-b014-9ddb293af6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz\" (UID: \"7bcd1531-3b66-4b65-b014-9ddb293af6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:55.708149 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.708123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd1531-3b66-4b65-b014-9ddb293af6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz\" (UID: \"7bcd1531-3b66-4b65-b014-9ddb293af6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:55.768964 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.768893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:55.909895 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.909856 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:49:55.922383 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:55.922340 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz"] Jul 12 17:49:55.923239 ip-10-0-139-89 kubenswrapper[2573]: W0712 17:49:55.923202 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcd1531_3b66_4b65_b014_9ddb293af6b4.slice/crio-8033dbb70fffd647458319da58f96cf90622e3ba188c386e1395f0d7d50dc057 WatchSource:0}: Error finding container 8033dbb70fffd647458319da58f96cf90622e3ba188c386e1395f0d7d50dc057: Status 404 returned error can't find the container with id 8033dbb70fffd647458319da58f96cf90622e3ba188c386e1395f0d7d50dc057 Jul 12 17:49:56.010422 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:56.010390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:49:56.373080 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:56.373040 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" event={"ID":"7bcd1531-3b66-4b65-b014-9ddb293af6b4","Type":"ContainerStarted","Data":"0efa523f22935c8debd443e2e11813fc1c9b1d000857d2f269a7bc8c3dad7edf"} Jul 12 17:49:56.373264 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:56.373103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" event={"ID":"7bcd1531-3b66-4b65-b014-9ddb293af6b4","Type":"ContainerStarted","Data":"8033dbb70fffd647458319da58f96cf90622e3ba188c386e1395f0d7d50dc057"} Jul 12 17:49:56.387900 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:56.387846 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" podStartSLOduration=1.387829331 podStartE2EDuration="1.387829331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:49:56.387280348 +0000 UTC m=+2803.882186789" watchObservedRunningTime="2026-07-12 17:49:56.387829331 +0000 UTC m=+2803.882735772" Jul 12 17:49:57.380767 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:57.380736 2573 generic.go:356] "Generic (PLEG): container finished" podID="7bcd1531-3b66-4b65-b014-9ddb293af6b4" containerID="0efa523f22935c8debd443e2e11813fc1c9b1d000857d2f269a7bc8c3dad7edf" exitCode=0 Jul 12 17:49:57.381199 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:57.380828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" event={"ID":"7bcd1531-3b66-4b65-b014-9ddb293af6b4","Type":"ContainerDied","Data":"0efa523f22935c8debd443e2e11813fc1c9b1d000857d2f269a7bc8c3dad7edf"} Jul 12 17:49:58.540474 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:58.540445 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:58.637601 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:58.637508 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd1531-3b66-4b65-b014-9ddb293af6b4-maas-api-service-ca\") pod \"7bcd1531-3b66-4b65-b014-9ddb293af6b4\" (UID: \"7bcd1531-3b66-4b65-b014-9ddb293af6b4\") " Jul 12 17:49:58.637876 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:58.637852 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcd1531-3b66-4b65-b014-9ddb293af6b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bcd1531-3b66-4b65-b014-9ddb293af6b4" (UID: "7bcd1531-3b66-4b65-b014-9ddb293af6b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:49:58.738997 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:58.738956 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd1531-3b66-4b65-b014-9ddb293af6b4-maas-api-service-ca\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:49:59.394621 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:59.394537 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" Jul 12 17:49:59.394621 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:59.394575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz" event={"ID":"7bcd1531-3b66-4b65-b014-9ddb293af6b4","Type":"ContainerDied","Data":"8033dbb70fffd647458319da58f96cf90622e3ba188c386e1395f0d7d50dc057"} Jul 12 17:49:59.394807 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:49:59.394645 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8033dbb70fffd647458319da58f96cf90622e3ba188c386e1395f0d7d50dc057" Jul 12 17:50:05.816868 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:05.816833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:10.534386 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:10.534337 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:14.021268 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:14.021231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:14.121533 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:14.121500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:16.433716 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:16.433682 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:16.535963 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:16.535894 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:21.243279 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:21.243240 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz"] Jul 12 17:50:21.243767 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:21.243498 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" containerID="cri-o://0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3" gracePeriod=30 Jul 12 17:50:21.256687 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:21.256640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:21.482619 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:21.482584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:22.158966 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:22.158887 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:22.260202 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:22.260169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:24.456287 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:24.456233 2573 patch_prober.go:28] interesting pod/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 17:50:24.456773 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:24.456305 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 17:50:26.512578 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.512512 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:50:26.552641 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.552597 2573 generic.go:356] "Generic (PLEG): container finished" podID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerID="0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3" exitCode=0 Jul 12 17:50:26.552829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.552639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" event={"ID":"3bfb805a-d0ef-4a57-87a4-04077636f13a","Type":"ContainerDied","Data":"0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3"} Jul 12 17:50:26.552829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.552658 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" Jul 12 17:50:26.552829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.552705 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz" event={"ID":"3bfb805a-d0ef-4a57-87a4-04077636f13a","Type":"ContainerDied","Data":"a5b1a59e7c7b41fa74d73f1c76adb5a91cfb7d7b9fb5f9e96220335a60362f1d"} Jul 12 17:50:26.552829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.552735 2573 scope.go:117] "RemoveContainer" containerID="0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3" Jul 12 17:50:26.562165 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.562149 2573 scope.go:117] "RemoveContainer" containerID="efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108" Jul 12 17:50:26.570756 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.570738 2573 scope.go:117] "RemoveContainer" containerID="0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3" Jul 12 17:50:26.571040 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:50:26.571022 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3\": container with ID starting with 0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3 not found: ID does not exist" containerID="0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3" Jul 12 17:50:26.571096 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.571052 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3"} err="failed to get container status \"0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3\": rpc error: code = NotFound desc = could not find container \"0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3\": container with ID starting with 0851442aa4d54eb8d0e9a6531de17420777896c25fe904e0cd7a5dd02ec650b3 not found: ID does not exist" Jul 12 17:50:26.571096 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.571072 2573 scope.go:117] "RemoveContainer" containerID="efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108" Jul 12 17:50:26.571292 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:50:26.571276 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108\": container with ID starting with efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108 not found: ID does not exist" containerID="efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108" Jul 12 17:50:26.571344 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.571294 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108"} err="failed to get container status \"efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108\": rpc error: code = NotFound desc = could not find container \"efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108\": container with ID starting with efe4c0c60ab0c845cf11b0a2c33896d527ae25a648d112feba8d6cac705c6108 not found: ID does not exist" Jul 12 17:50:26.618496 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618455 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-socket\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.618705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618514 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-token\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.618705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618545 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-podinfo\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.618705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618579 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-data\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.618705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618605 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-envoy\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.618705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618644 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-credential-socket\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.618705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618694 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-certs\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.619057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618730 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfb805a-d0ef-4a57-87a4-04077636f13a-istiod-ca-cert\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.619057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618762 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s65kv\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-kube-api-access-s65kv\") pod \"3bfb805a-d0ef-4a57-87a4-04077636f13a\" (UID: \"3bfb805a-d0ef-4a57-87a4-04077636f13a\") " Jul 12 17:50:26.619057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618802 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:26.619057 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.618988 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:26.619287 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.619169 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfb805a-d0ef-4a57-87a4-04077636f13a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:50:26.619287 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.619234 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.619287 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.619256 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.619468 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.619418 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:26.619596 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.619571 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:26.620955 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.620898 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:26.621081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.620981 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:50:26.621081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.620996 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:50:26.621081 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.621064 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-kube-api-access-s65kv" (OuterVolumeSpecName: "kube-api-access-s65kv") pod "3bfb805a-d0ef-4a57-87a4-04077636f13a" (UID: "3bfb805a-d0ef-4a57-87a4-04077636f13a"). InnerVolumeSpecName "kube-api-access-s65kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:50:26.720581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720551 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.720581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720580 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfb805a-d0ef-4a57-87a4-04077636f13a-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.720581 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720589 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65kv\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-kube-api-access-s65kv\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.720813 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720599 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.720813 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720608 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.720813 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720616 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.720813 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.720623 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfb805a-d0ef-4a57-87a4-04077636f13a-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:26.874902 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.874866 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz"] Jul 12 17:50:26.878957 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:26.878931 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e767e8-openshift-default-6d549747f-cvdtz"] Jul 12 17:50:27.039801 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:27.039769 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" path="/var/lib/kubelet/pods/3bfb805a-d0ef-4a57-87a4-04077636f13a/volumes" Jul 12 17:50:31.996473 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:31.996433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:36.695144 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:36.695101 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:42.420423 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:42.420381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:42.477986 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:42.477934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:47.346904 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:47.346864 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972"] Jul 12 17:50:47.347528 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:47.347169 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" containerID="cri-o://7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21" gracePeriod=30 Jul 12 17:50:47.372121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:47.372085 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:47.594709 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:47.594672 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:48.024075 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:48.024038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69c5776576-5rjdn_1835fe20-3610-433c-b738-0195d1541d3e/authorino/0.log" Jul 12 17:50:50.157385 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:50.157332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:50.206366 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:50.206329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:50:52.594428 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.594406 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:50:52.671003 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.670964 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-socket\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-credential-socket\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671090 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-envoy\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671118 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-data\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671138 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-podinfo\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671155 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-certs\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671193 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istiod-ca-cert\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671523 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671218 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gn54\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-kube-api-access-7gn54\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671523 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671264 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-token\") pod \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\" (UID: \"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71\") " Jul 12 17:50:52.671523 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671448 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:52.671691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671507 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:52.671691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671614 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 17:50:52.671691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671620 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-credential-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.671691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671669 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-certs\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.671691 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.671674 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:52.672299 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.672270 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:52.673303 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.673285 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:50:52.673419 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.673399 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 17:50:52.673844 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.673821 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-kube-api-access-7gn54" (OuterVolumeSpecName: "kube-api-access-7gn54") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "kube-api-access-7gn54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:50:52.673969 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.673845 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" (UID: "1e5bf9bb-0276-46c9-a85c-c96cf8f18b71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:50:52.704163 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.704129 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerID="7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21" exitCode=0 Jul 12 17:50:52.704298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.704181 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" event={"ID":"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71","Type":"ContainerDied","Data":"7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21"} Jul 12 17:50:52.704298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.704227 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" event={"ID":"1e5bf9bb-0276-46c9-a85c-c96cf8f18b71","Type":"ContainerDied","Data":"60e9c160c9166a01ad1f415a6669d5d28091fb80a08e9f0abdb96878770d99f0"} Jul 12 17:50:52.704298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.704260 2573 scope.go:117] "RemoveContainer" containerID="7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21" Jul 12 17:50:52.704434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.704186 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972" Jul 12 17:50:52.714707 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.714685 2573 scope.go:117] "RemoveContainer" containerID="96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee" Jul 12 17:50:52.724588 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.724567 2573 scope.go:117] "RemoveContainer" containerID="7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21" Jul 12 17:50:52.724949 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:50:52.724930 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21\": container with ID starting with 7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21 not found: ID does not exist" containerID="7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21" Jul 12 17:50:52.725015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.724959 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21"} err="failed to get container status \"7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21\": rpc error: code = NotFound desc = could not find container \"7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21\": container with ID starting with 7e44b5c6ce0400d96ebd40631ebd688d0c04602ad48f53d37db2948632bf6f21 not found: ID does not exist" Jul 12 17:50:52.725015 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.724977 2573 scope.go:117] "RemoveContainer" containerID="96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee" Jul 12 17:50:52.725222 ip-10-0-139-89 kubenswrapper[2573]: E0712 17:50:52.725203 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee\": container with ID starting with 96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee not found: ID does not exist" containerID="96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee" Jul 12 17:50:52.725265 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.725228 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee"} err="failed to get container status \"96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee\": rpc error: code = NotFound desc = could not find container \"96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee\": container with ID starting with 96d1cf97b1871c5ef41cea4a0778dc1f34342a567566a931bc3f52b32c9e45ee not found: ID does not exist" Jul 12 17:50:52.726298 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.726274 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972"] Jul 12 17:50:52.729907 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.729875 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-9c75b6-openshift-default-76d55fd8d7-l8972"] Jul 12 17:50:52.772607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772573 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-token\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.772607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772603 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-workload-socket\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.772607 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772612 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-envoy\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.772847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772622 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-data\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.772847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772632 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istio-podinfo\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.772847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772642 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-istiod-ca-cert\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:52.772847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:52.772650 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gn54\" (UniqueName: \"kubernetes.io/projected/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71-kube-api-access-7gn54\") on node \"ip-10-0-139-89.ec2.internal\" DevicePath \"\"" Jul 12 17:50:53.039268 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:53.039168 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" path="/var/lib/kubelet/pods/1e5bf9bb-0276-46c9-a85c-c96cf8f18b71/volumes" Jul 12 17:50:53.290669 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:53.290577 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-x2kwh_be76c4d0-6b33-45a8-b8f4-ec42379b8e70/manager/0.log" Jul 12 17:50:53.418930 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:53.418874 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cll2h_a0f6be29-8606-4180-aec2-5311c9416e90/manager/0.log" Jul 12 17:50:53.558225 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:53.558114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b576ccbb6-csl7p_3f1cda9a-eedb-425e-9650-11c3997379dd/manager/0.log" Jul 12 17:50:53.684244 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:53.684210 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pj8pm_ecb488a1-c450-4b2d-91eb-8f54f4c118c2/server/0.log" Jul 12 17:50:53.819982 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:53.819872 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b6nbq_f94e4be4-b2df-48d2-824c-4a136d96aca4/manager/0.log" Jul 12 17:50:54.237949 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:54.237898 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-hnhv6_408afd12-f329-4b1e-9228-4d40fc53d53a/copy-manifests/0.log" Jul 12 17:50:54.256068 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:54.256027 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-hnhv6_408afd12-f329-4b1e-9228-4d40fc53d53a/manager/0.log" Jul 12 17:50:54.675972 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:54.675940 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6db56bb667-z6tkj_7acea634-f4c9-411a-a3e6-be310b9ac0bb/maas-api/0.log" Jul 12 17:50:54.791019 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:54.790990 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-187a6623-jl9jk_b51b0fd0-2710-4110-849f-21c5cd205ee3/revoke-keys/1.log" Jul 12 17:50:54.905313 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:54.905285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-2c119f77-k97wp_18fa55b9-4177-454e-acea-0edee3da3f88/revoke-keys/0.log" Jul 12 17:50:55.016933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:55.016825 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-444d10-7hmnw_9a70c23e-b388-4667-8b94-ab4bf69ff144/revoke-keys/0.log" Jul 12 17:50:55.586985 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:55.586946 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-dc65b9aa-sfbwn_fa7253a4-9dfb-49cb-a434-c8e53fec6890/revoke-keys/0.log" Jul 12 17:50:55.967596 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:55.967566 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-e767e8-tgjrz_7bcd1531-3b66-4b65-b014-9ddb293af6b4/revoke-keys/0.log" Jul 12 17:50:56.193927 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:56.193884 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bsczp_d56491b7-3ea2-46e7-8f57-4124f6767521/postgres/0.log" Jul 12 17:50:57.517650 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:57.517620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69c5776576-5rjdn_1835fe20-3610-433c-b738-0195d1541d3e/authorino/0.log" Jul 12 17:50:57.643367 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:57.643331 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5hcqw_fa51c4cf-ab54-439e-a656-a7f2cfa06e6b/manager/0.log" Jul 12 17:50:57.756876 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:57.756847 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7nlfs_45a23379-cdf6-4b51-b79f-a3945004016a/manager/0.log" Jul 12 17:50:57.994121 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:57.994093 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-qskq6_7b61b4a9-bc14-4bff-9266-ad5c5b962c38/registry-server/0.log" Jul 12 17:50:58.139229 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:58.139189 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-97kpn_2c1d50c1-beb6-4f98-96b2-e882ce6094b6/manager/0.log" Jul 12 17:50:58.391705 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:58.391624 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-b4tp2_1814b74e-9cd4-4d70-ac44-86f5614fc31e/manager/0.log" Jul 12 17:50:59.062559 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:59.062523 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-wlcj8_659b6479-271a-45c8-9a9f-8c1303778839/kube-auth-proxy/0.log" Jul 12 17:50:59.299392 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:59.299360 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-k9cws_f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a/payload-pre-processing/0.log" Jul 12 17:50:59.420172 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:50:59.420135 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-4qp9w_a82263e2-db57-4511-9537-7d212c8bf5ea/payload-processing/0.log" Jul 12 17:51:00.198764 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:00.198733 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbh7mc2_8585d4b6-5d3b-40a0-9d2a-4c2350e9b950/main/0.log" Jul 12 17:51:04.579123 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579081 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rpjth/must-gather-9mgfb"] Jul 12 17:51:04.579717 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579693 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" Jul 12 17:51:04.579863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579793 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" Jul 12 17:51:04.579863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579821 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" Jul 12 17:51:04.579863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579830 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" Jul 12 17:51:04.579863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579848 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" Jul 12 17:51:04.579863 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579857 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579871 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcd1531-3b66-4b65-b014-9ddb293af6b4" containerName="revoke-keys" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.579880 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcd1531-3b66-4b65-b014-9ddb293af6b4" containerName="revoke-keys" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580046 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580063 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580072 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcd1531-3b66-4b65-b014-9ddb293af6b4" containerName="revoke-keys" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580084 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" Jul 12 17:51:04.580185 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580095 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfb805a-d0ef-4a57-87a4-04077636f13a" containerName="istio-proxy" Jul 12 17:51:04.580518 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580215 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" Jul 12 17:51:04.580518 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.580226 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5bf9bb-0276-46c9-a85c-c96cf8f18b71" containerName="istio-proxy" Jul 12 17:51:04.583537 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.583517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.586092 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.586073 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpjth\"/\"kube-root-ca.crt\"" Jul 12 17:51:04.587079 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.587046 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rpjth\"/\"default-dockercfg-pfmzn\"" Jul 12 17:51:04.587079 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.587057 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpjth\"/\"openshift-service-ca.crt\"" Jul 12 17:51:04.591005 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.590976 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpjth/must-gather-9mgfb"] Jul 12 17:51:04.603285 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.603243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:51:04.659389 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.659310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" limit=1024 Jul 12 17:51:04.687119 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.687087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27st\" (UniqueName: \"kubernetes.io/projected/df27e1d2-0aa3-417b-a0a8-40a8cbabce87-kube-api-access-g27st\") pod \"must-gather-9mgfb\" (UID: \"df27e1d2-0aa3-417b-a0a8-40a8cbabce87\") " pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.687282 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.687124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/df27e1d2-0aa3-417b-a0a8-40a8cbabce87-must-gather-output\") pod \"must-gather-9mgfb\" (UID: \"df27e1d2-0aa3-417b-a0a8-40a8cbabce87\") " pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.788502 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.788467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27st\" (UniqueName: \"kubernetes.io/projected/df27e1d2-0aa3-417b-a0a8-40a8cbabce87-kube-api-access-g27st\") pod \"must-gather-9mgfb\" (UID: \"df27e1d2-0aa3-417b-a0a8-40a8cbabce87\") " pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.788502 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.788502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/df27e1d2-0aa3-417b-a0a8-40a8cbabce87-must-gather-output\") pod \"must-gather-9mgfb\" (UID: \"df27e1d2-0aa3-417b-a0a8-40a8cbabce87\") " pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.788839 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.788800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/df27e1d2-0aa3-417b-a0a8-40a8cbabce87-must-gather-output\") pod \"must-gather-9mgfb\" (UID: \"df27e1d2-0aa3-417b-a0a8-40a8cbabce87\") " pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.796010 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.795982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27st\" (UniqueName: \"kubernetes.io/projected/df27e1d2-0aa3-417b-a0a8-40a8cbabce87-kube-api-access-g27st\") pod \"must-gather-9mgfb\" (UID: \"df27e1d2-0aa3-417b-a0a8-40a8cbabce87\") " pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:04.894605 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:04.894513 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpjth/must-gather-9mgfb" Jul 12 17:51:05.042257 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:05.042230 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpjth/must-gather-9mgfb"] Jul 12 17:51:05.781187 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:05.781149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/must-gather-9mgfb" event={"ID":"df27e1d2-0aa3-417b-a0a8-40a8cbabce87","Type":"ContainerStarted","Data":"c0a54906803a05cf3f6a608d8035e390b0e6a8bdec6b0addb8ce5af53fa83e20"} Jul 12 17:51:06.792176 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:06.792125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/must-gather-9mgfb" event={"ID":"df27e1d2-0aa3-417b-a0a8-40a8cbabce87","Type":"ContainerStarted","Data":"34969d6337aee7010b58fe81b6d0872b119129f4b812cbf652eab5b53130bc37"} Jul 12 17:51:06.792176 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:06.792177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/must-gather-9mgfb" event={"ID":"df27e1d2-0aa3-417b-a0a8-40a8cbabce87","Type":"ContainerStarted","Data":"94f6bdb68d837f4f128a1f1c567711badd2027b969673fecdbc5175e15d584a9"} Jul 12 17:51:06.808557 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:06.808485 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rpjth/must-gather-9mgfb" podStartSLOduration=1.981042105 podStartE2EDuration="2.808463809s" totalImagesPullingTime="827.421704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:51:06.805873317 +0000 UTC m=+2874.300779757" watchObservedRunningTime="2026-07-12 17:51:06.808463809 +0000 UTC m=+2874.303370250" Jul 12 17:51:07.431667 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:07.431635 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tw7w7_ef941278-2d82-4baa-bb36-caa777b09ab0/global-pull-secret-syncer/0.log" Jul 12 17:51:07.527144 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:07.527107 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4qprc_8a35afdb-8438-4880-a499-ff342dee3802/konnectivity-agent/0.log" Jul 12 17:51:07.636050 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:07.636015 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-89.ec2.internal_bafc608fa264eba1d850d33cd5b6db88/haproxy/0.log" Jul 12 17:51:13.100847 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:13.100811 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69c5776576-5rjdn_1835fe20-3610-433c-b738-0195d1541d3e/authorino/0.log" Jul 12 17:51:13.154011 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:13.153980 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5hcqw_fa51c4cf-ab54-439e-a656-a7f2cfa06e6b/manager/0.log" Jul 12 17:51:13.180495 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:13.180450 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7nlfs_45a23379-cdf6-4b51-b79f-a3945004016a/manager/0.log" Jul 12 17:51:13.258943 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:13.258894 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-qskq6_7b61b4a9-bc14-4bff-9266-ad5c5b962c38/registry-server/0.log" Jul 12 17:51:13.486320 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:13.486282 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-97kpn_2c1d50c1-beb6-4f98-96b2-e882ce6094b6/manager/0.log" Jul 12 17:51:13.607936 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:13.607868 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-b4tp2_1814b74e-9cd4-4d70-ac44-86f5614fc31e/manager/0.log" Jul 12 17:51:15.245829 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.245795 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/alertmanager/0.log" Jul 12 17:51:15.267694 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.267668 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/config-reloader/0.log" Jul 12 17:51:15.286584 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.286542 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/kube-rbac-proxy-web/0.log" Jul 12 17:51:15.314118 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.314089 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/kube-rbac-proxy/0.log" Jul 12 17:51:15.333403 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.333378 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/kube-rbac-proxy-metric/0.log" Jul 12 17:51:15.354056 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.354024 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/prom-label-proxy/0.log" Jul 12 17:51:15.394855 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.374121 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af34459a-a7f4-4e1e-9516-6450972d1240/init-config-reloader/0.log" Jul 12 17:51:15.473142 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.473106 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-49k5k_bafc58cf-1e74-4ce0-8907-252eebde85c6/kube-state-metrics/0.log" Jul 12 17:51:15.489424 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.489387 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-49k5k_bafc58cf-1e74-4ce0-8907-252eebde85c6/kube-rbac-proxy-main/0.log" Jul 12 17:51:15.511969 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.511862 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-49k5k_bafc58cf-1e74-4ce0-8907-252eebde85c6/kube-rbac-proxy-self/0.log" Jul 12 17:51:15.820362 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.820272 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbw2r_3d84b57e-568b-4ce9-b514-5b348444bc66/node-exporter/0.log" Jul 12 17:51:15.841626 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.841591 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbw2r_3d84b57e-568b-4ce9-b514-5b348444bc66/kube-rbac-proxy/0.log" Jul 12 17:51:15.866560 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:15.866530 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbw2r_3d84b57e-568b-4ce9-b514-5b348444bc66/init-textfile/0.log" Jul 12 17:51:16.202938 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.202872 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fbbfd5d5c-qj7hq_cefdbe10-39f7-4b6c-8c60-e9230740c6bc/telemeter-client/0.log" Jul 12 17:51:16.221649 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.221617 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fbbfd5d5c-qj7hq_cefdbe10-39f7-4b6c-8c60-e9230740c6bc/reload/0.log" Jul 12 17:51:16.240449 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.240415 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fbbfd5d5c-qj7hq_cefdbe10-39f7-4b6c-8c60-e9230740c6bc/kube-rbac-proxy/0.log" Jul 12 17:51:16.270114 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.270083 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f764f5c4-rsj95_d009e2f0-2268-4d72-b83f-8a53484b8b81/thanos-query/0.log" Jul 12 17:51:16.288933 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.288880 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f764f5c4-rsj95_d009e2f0-2268-4d72-b83f-8a53484b8b81/kube-rbac-proxy-web/0.log" Jul 12 17:51:16.308728 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.308696 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f764f5c4-rsj95_d009e2f0-2268-4d72-b83f-8a53484b8b81/kube-rbac-proxy/0.log" Jul 12 17:51:16.327140 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.327109 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f764f5c4-rsj95_d009e2f0-2268-4d72-b83f-8a53484b8b81/prom-label-proxy/0.log" Jul 12 17:51:16.345827 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.345796 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f764f5c4-rsj95_d009e2f0-2268-4d72-b83f-8a53484b8b81/kube-rbac-proxy-rules/0.log" Jul 12 17:51:16.364342 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:16.364309 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f764f5c4-rsj95_d009e2f0-2268-4d72-b83f-8a53484b8b81/kube-rbac-proxy-metrics/0.log" Jul 12 17:51:17.782116 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.782079 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd"] Jul 12 17:51:17.793228 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.793196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.794254 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.794227 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd"] Jul 12 17:51:17.847104 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.847071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-lib-modules\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.847441 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.847418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-proc\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.847591 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.847575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-sys\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.847732 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.847702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wk5\" (UniqueName: \"kubernetes.io/projected/0925444e-fc17-4b95-b643-73219f35c85d-kube-api-access-c9wk5\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.847899 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.847886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-podres\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.949731 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.949697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-lib-modules\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950182 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-proc\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950332 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-lib-modules\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950420 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-proc\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950484 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-sys\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950545 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wk5\" (UniqueName: \"kubernetes.io/projected/0925444e-fc17-4b95-b643-73219f35c85d-kube-api-access-c9wk5\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950598 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-podres\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950707 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-sys\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.950790 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.950740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0925444e-fc17-4b95-b643-73219f35c85d-podres\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:17.959063 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:17.959039 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wk5\" (UniqueName: \"kubernetes.io/projected/0925444e-fc17-4b95-b643-73219f35c85d-kube-api-access-c9wk5\") pod \"perf-node-gather-daemonset-rn9pd\" (UID: \"0925444e-fc17-4b95-b643-73219f35c85d\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:18.108303 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.108219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:18.303224 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.302869 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd"] Jul 12 17:51:18.696024 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.695960 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c66b7b8bf-gqvfm_914ef91a-2af5-4803-88a9-fb54cbae72e7/console/0.log" Jul 12 17:51:18.904108 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.904059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" event={"ID":"0925444e-fc17-4b95-b643-73219f35c85d","Type":"ContainerStarted","Data":"3a3d65eae77a9039373997e8f24f37bf07f26cd4d78dcb851aae3a77fb164c80"} Jul 12 17:51:18.904533 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.904144 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" event={"ID":"0925444e-fc17-4b95-b643-73219f35c85d","Type":"ContainerStarted","Data":"1330833221bed9286edc155fe1eb7bfccacc61d4f165e75fe18b7c86ed664b48"} Jul 12 17:51:18.904533 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.904255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:18.919391 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:18.919333 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" podStartSLOduration=1.919293489 podStartE2EDuration="1.919293489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:51:18.91823602 +0000 UTC m=+2886.413142460" watchObservedRunningTime="2026-07-12 17:51:18.919293489 +0000 UTC m=+2886.414199929" Jul 12 17:51:20.379192 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:20.379152 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wd2sd_a2b9bcb2-7f37-4544-a569-eca348e5e03b/dns/0.log" Jul 12 17:51:20.399110 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:20.399076 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wd2sd_a2b9bcb2-7f37-4544-a569-eca348e5e03b/kube-rbac-proxy/0.log" Jul 12 17:51:20.464113 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:20.464081 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jk4d_1223e5b6-e4d5-4c8d-bf67-e3723e46ea51/dns-node-resolver/0.log" Jul 12 17:51:21.016539 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:21.016489 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9cc8fc59b-t2mh4_9db122e2-3b4a-4d3f-b533-eb1ce590b4b3/registry/0.log" Jul 12 17:51:21.088138 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:21.088102 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x48ln_00c5aa6e-c15c-4b8a-b201-44c4355ef991/node-ca/0.log" Jul 12 17:51:22.114201 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:22.114167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-wlcj8_659b6479-271a-45c8-9a9f-8c1303778839/kube-auth-proxy/0.log" Jul 12 17:51:22.206620 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:22.206593 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-k9cws_f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a/payload-pre-processing/0.log" Jul 12 17:51:22.244485 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:22.244451 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-4qp9w_a82263e2-db57-4511-9537-7d212c8bf5ea/payload-processing/0.log" Jul 12 17:51:22.966679 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:22.966648 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-df2gd_1f22e4c8-ca08-4c16-80f2-c31d9eb24d70/serve-healthcheck-canary/0.log" Jul 12 17:51:23.444894 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:23.444863 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wctm5_27e62b21-0b33-483d-8806-430a8ebb21c5/kube-rbac-proxy/0.log" Jul 12 17:51:23.464013 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:23.463986 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wctm5_27e62b21-0b33-483d-8806-430a8ebb21c5/exporter/0.log" Jul 12 17:51:23.484690 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:23.484662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wctm5_27e62b21-0b33-483d-8806-430a8ebb21c5/extractor/0.log" Jul 12 17:51:24.924867 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:24.924839 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-rn9pd" Jul 12 17:51:25.633434 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:25.633390 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-x2kwh_be76c4d0-6b33-45a8-b8f4-ec42379b8e70/manager/0.log" Jul 12 17:51:25.882984 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:25.882954 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cll2h_a0f6be29-8606-4180-aec2-5311c9416e90/manager/0.log" Jul 12 17:51:26.086222 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:26.086180 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b576ccbb6-csl7p_3f1cda9a-eedb-425e-9650-11c3997379dd/manager/0.log" Jul 12 17:51:26.109151 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:26.109125 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pj8pm_ecb488a1-c450-4b2d-91eb-8f54f4c118c2/server/0.log" Jul 12 17:51:26.214495 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:26.214462 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b6nbq_f94e4be4-b2df-48d2-824c-4a136d96aca4/manager/0.log" Jul 12 17:51:26.425999 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:26.425957 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-hnhv6_408afd12-f329-4b1e-9228-4d40fc53d53a/manager/0.log" Jul 12 17:51:26.444538 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:26.444508 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-hnhv6_408afd12-f329-4b1e-9228-4d40fc53d53a/copy-manifests/0.log" Jul 12 17:51:27.862707 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:27.862620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-74phb_89eb31b7-b4f9-4f54-9caa-2e1b27d4cdad/openshift-lws-operator/0.log" Jul 12 17:51:28.879112 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:28.879081 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-wlcj8_659b6479-271a-45c8-9a9f-8c1303778839/kube-auth-proxy/0.log" Jul 12 17:51:28.969639 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:28.969606 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-k9cws_f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a/payload-pre-processing/0.log" Jul 12 17:51:28.997294 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:28.997257 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-4qp9w_a82263e2-db57-4511-9537-7d212c8bf5ea/payload-processing/0.log" Jul 12 17:51:30.428515 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:30.428473 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-wlcj8_659b6479-271a-45c8-9a9f-8c1303778839/kube-auth-proxy/0.log" Jul 12 17:51:30.502289 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:30.502246 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-k9cws_f60ac4cb-9e42-41ff-b1bf-e7e72fccf66a/payload-pre-processing/0.log" Jul 12 17:51:30.533608 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:30.533576 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-4qp9w_a82263e2-db57-4511-9537-7d212c8bf5ea/payload-processing/0.log" Jul 12 17:51:34.274712 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.274675 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-58k2l_cf74a844-f099-4a72-804e-0106c3b44a3c/kube-multus/0.log" Jul 12 17:51:34.589366 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.589276 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/kube-multus-additional-cni-plugins/0.log" Jul 12 17:51:34.608630 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.608600 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/egress-router-binary-copy/0.log" Jul 12 17:51:34.626403 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.626375 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/cni-plugins/0.log" Jul 12 17:51:34.644423 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.644385 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/bond-cni-plugin/0.log" Jul 12 17:51:34.662086 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.662057 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/routeoverride-cni/0.log" Jul 12 17:51:34.679960 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.679904 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/whereabouts-cni-bincopy/0.log" Jul 12 17:51:34.702321 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.702290 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v79f6_eab4f6c6-ddc2-4337-aa2b-e1f41e811818/whereabouts-cni/0.log" Jul 12 17:51:34.827011 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.826898 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-clc6z_7115496d-2b24-4f09-b65a-678040e4a303/network-metrics-daemon/0.log" Jul 12 17:51:34.841356 ip-10-0-139-89 kubenswrapper[2573]: I0712 17:51:34.841283 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-clc6z_7115496d-2b24-4f09-b65a-678040e4a303/kube-rbac-proxy/0.log"