Jul 08 02:40:09.603631 ip-10-0-128-107 systemd[1]: Starting Kubernetes Kubelet... Jul 08 02:40:10.006461 ip-10-0-128-107 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:40:10.006461 ip-10-0-128-107 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 02:40:10.006461 ip-10-0-128-107 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:40:10.006461 ip-10-0-128-107 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 02:40:10.006461 ip-10-0-128-107 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:40:10.009323 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.009257 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 02:40:10.013562 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013547 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:40:10.013562 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013561 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013565 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013568 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013571 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013574 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013577 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013581 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013586 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013588 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013591 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013594 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013597 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013600 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013602 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013605 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013608 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013610 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013612 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013615 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:40:10.013627 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013617 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013620 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013622 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013625 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013628 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013630 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013633 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013636 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013639 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013641 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013644 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013648 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013653 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013656 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013658 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013661 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013664 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013667 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013669 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:40:10.014076 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013672 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013674 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013678 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013681 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013684 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013687 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013689 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013692 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013694 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013697 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013699 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013701 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013704 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013706 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013708 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013711 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013714 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013717 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013719 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:40:10.014558 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013721 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013724 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013726 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013729 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013731 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013734 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013736 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013739 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013741 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013744 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013746 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013749 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013751 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013754 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013757 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013759 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013762 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013764 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013767 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013769 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:40:10.015051 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013771 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013774 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013777 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013779 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013781 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013784 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013786 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013788 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013792 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013795 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013802 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013805 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013808 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013810 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013813 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013815 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013818 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013820 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013822 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013825 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:40:10.015553 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.013828 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015931 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015939 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015942 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015946 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015949 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015952 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015955 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015957 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015960 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015962 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015965 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015967 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015970 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015972 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015975 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015978 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015980 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015982 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015985 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:40:10.016117 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015987 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015990 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015992 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015995 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015997 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.015999 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016002 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016004 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016006 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016010 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016013 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016016 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016019 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016022 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016025 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016028 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016031 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016033 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016036 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:40:10.016620 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016038 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016041 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016043 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016046 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016049 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016051 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016054 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016056 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016058 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016061 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016063 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016065 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016068 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016070 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016074 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016078 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016081 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016084 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016087 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016089 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:40:10.017098 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016092 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016094 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016097 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016099 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016102 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016104 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016107 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016110 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016113 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016115 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016118 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016121 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016123 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016126 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016129 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016131 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016134 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016136 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016139 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016141 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:40:10.017597 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016144 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016146 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016148 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016152 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016155 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016158 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016161 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016164 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016166 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016169 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016172 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016175 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016178 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016180 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016182 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016185 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016188 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016190 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016193 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016196 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:40:10.018087 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016198 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016584 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016589 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016592 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016595 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016598 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016601 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016604 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016606 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016609 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016612 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016614 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016617 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016619 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016622 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016625 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016627 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016630 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016632 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016635 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:40:10.018588 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016637 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016641 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016643 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016646 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016648 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016651 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016653 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016656 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016658 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016661 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016663 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016666 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016669 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016672 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016674 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016676 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016679 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016681 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016684 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016687 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:40:10.019071 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016690 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016693 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016697 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016700 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016702 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016705 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016707 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016710 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016714 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016717 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016720 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016722 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016725 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016728 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016731 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016733 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016736 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016738 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016741 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:40:10.019572 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016744 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016747 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016749 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016751 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016754 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016757 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016759 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016761 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016764 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016766 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016769 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016772 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016774 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016777 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016779 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016782 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016784 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016787 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016789 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:40:10.020066 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016792 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016795 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016797 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016800 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016802 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016805 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016807 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016811 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016814 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016817 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016819 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016822 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016824 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016827 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016829 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016833 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016836 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016839 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016842 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:40:10.020594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016844 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016847 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.016849 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016919 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016926 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016931 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016935 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016940 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016943 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016947 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016951 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016954 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016957 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016960 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016964 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016967 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016970 2576 flags.go:64] FLAG: --cgroup-root="" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016972 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016975 2576 flags.go:64] FLAG: --client-ca-file="" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016978 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016982 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016987 2576 flags.go:64] FLAG: --cluster-domain="" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016990 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016992 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 02:40:10.021124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016996 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.016999 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017002 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017005 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017008 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017011 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017014 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017017 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017020 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017023 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017026 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017030 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017033 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017035 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017038 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017041 2576 flags.go:64] FLAG: --enable-server="true" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017059 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017064 2576 flags.go:64] FLAG: --event-burst="100" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017067 2576 flags.go:64] FLAG: --event-qps="50" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017070 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017073 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017075 2576 flags.go:64] FLAG: --eviction-hard="" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017079 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017082 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017084 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 02:40:10.021792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017087 2576 flags.go:64] FLAG: --eviction-soft="" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017090 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017092 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017097 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017100 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017102 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017105 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017109 2576 flags.go:64] FLAG: --feature-gates="" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017113 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017115 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017118 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017121 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017124 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017127 2576 flags.go:64] FLAG: --help="false" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017130 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017132 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017135 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017138 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017141 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017144 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017146 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017149 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017152 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 02:40:10.022465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017154 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017157 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017160 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017163 2576 flags.go:64] FLAG: --kube-reserved="" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017166 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017168 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017171 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017173 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017176 2576 flags.go:64] FLAG: --lock-file="" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017179 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017181 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017184 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017191 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017194 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017196 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017199 2576 flags.go:64] FLAG: --logging-format="text" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017203 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017206 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017209 2576 flags.go:64] FLAG: --manifest-url="" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017212 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017216 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017218 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017222 2576 flags.go:64] FLAG: --max-pods="110" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017225 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017228 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 02:40:10.023121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017231 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017233 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017236 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017239 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017242 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017248 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017251 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017254 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017256 2576 flags.go:64] FLAG: --pod-cidr="" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017260 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017265 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017268 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017271 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017273 2576 flags.go:64] FLAG: --port="10250" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017276 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017279 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0efe078c5602e1a56" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017282 2576 flags.go:64] FLAG: --qos-reserved="" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017285 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017288 2576 flags.go:64] FLAG: --register-node="true" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017292 2576 flags.go:64] FLAG: --register-with-taints="" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017295 2576 flags.go:64] FLAG: --registry-burst="10" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017298 2576 flags.go:64] FLAG: --registry-qps="5" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017300 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017303 2576 flags.go:64] FLAG: --reserved-memory="" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017308 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 02:40:10.023928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017311 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017314 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017316 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017319 2576 flags.go:64] FLAG: --runonce="false" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017321 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017324 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017327 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017330 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017332 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017335 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017338 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017341 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017343 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017346 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017348 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017365 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017369 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017372 2576 flags.go:64] FLAG: --system-cgroups="" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017374 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017380 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017382 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017385 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017389 2576 flags.go:64] FLAG: --tls-min-version="" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017392 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017394 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 02:40:10.024613 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017397 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017401 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017404 2576 flags.go:64] FLAG: --v="2" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017408 2576 flags.go:64] FLAG: --version="false" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017411 2576 flags.go:64] FLAG: --vmodule="" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017415 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.017420 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017528 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017531 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017535 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017537 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017540 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017542 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017545 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017549 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017553 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017556 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017558 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017561 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017564 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017566 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:40:10.025226 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017569 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017572 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017575 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017577 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017580 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017582 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017585 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017587 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017589 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017592 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017594 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017597 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017601 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017604 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017606 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017609 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017612 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017615 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017618 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017620 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:40:10.025762 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017623 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017625 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017627 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017630 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017632 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017635 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017637 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017639 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017642 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017644 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017647 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017649 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017651 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017654 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017656 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017659 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017661 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017664 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017667 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017669 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:40:10.026255 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017672 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017674 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017677 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017679 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017682 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017684 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017686 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017690 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017693 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017696 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017699 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017701 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017703 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017706 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017708 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017710 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017713 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017715 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017718 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017720 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:40:10.026765 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017722 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017725 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017727 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017729 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017732 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017734 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017736 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017739 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017741 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017745 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017748 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017751 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017753 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017756 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017758 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017761 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017764 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017767 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017769 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:40:10.027429 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017771 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017774 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017776 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017779 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017781 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.017784 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.018441 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.024704 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.024718 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.025492 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.025499 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025553 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025558 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025562 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025565 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:40:10.028151 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025568 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025571 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025574 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025576 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025579 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025581 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025584 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025586 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025589 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025591 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025594 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025597 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025600 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025602 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025605 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025607 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025610 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025612 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025615 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025617 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:40:10.028783 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025620 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025622 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025624 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025627 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025630 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025633 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025636 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025640 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025643 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025647 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025649 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025652 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025655 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025658 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025660 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025663 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025666 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025669 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025672 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025674 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:40:10.029610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025677 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025679 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025682 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025685 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025687 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025690 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025693 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025695 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025697 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025700 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025702 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025706 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025709 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025711 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025714 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025717 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025720 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025723 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025726 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025729 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:40:10.030112 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025731 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025734 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025736 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025739 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025741 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025744 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025748 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025751 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025754 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025756 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025759 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025761 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025764 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025767 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025769 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025772 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025775 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025778 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025780 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:40:10.030822 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025783 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025785 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025787 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025790 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025792 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025795 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025797 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025799 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025802 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025804 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025807 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025810 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025813 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025815 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025818 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:40:10.031322 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025820 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.025825 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025934 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025938 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025942 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025946 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025949 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025952 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025956 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025960 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025962 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025965 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025968 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025970 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:40:10.031716 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025972 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025975 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025978 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025980 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025983 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025985 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025988 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025991 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025993 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025996 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.025998 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026001 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026003 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026006 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026008 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026011 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026014 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026017 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026019 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026022 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:40:10.032122 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026024 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026026 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026029 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026031 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026034 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026036 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026038 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026041 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026043 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026046 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026048 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026050 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026053 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026055 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026057 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026060 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026063 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026065 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026068 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026070 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:40:10.032633 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026072 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026075 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026077 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026080 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026082 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026084 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026086 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026090 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026093 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026096 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026099 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026102 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026104 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026107 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026109 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026111 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026114 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026116 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026118 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:40:10.033167 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026121 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026123 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026125 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026128 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026130 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026132 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026135 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026137 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026140 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026142 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026144 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026147 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026149 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026151 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026154 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026156 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026159 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026161 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026163 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:40:10.033677 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026166 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026168 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026170 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026173 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026175 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026178 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026180 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026182 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:10.026185 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.026189 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.026287 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.030290 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.031837 2576 server.go:1019] "Starting client certificate rotation" Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.031935 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 02:40:10.034159 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.031964 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 02:40:10.034827 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.034811 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 02:40:10.036028 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.036010 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 02:40:10.043224 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.043208 2576 log.go:25] "Validated CRI v1 runtime API" Jul 08 02:40:10.045166 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.045152 2576 log.go:25] "Validated CRI v1 image API" Jul 08 02:40:10.047559 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.047546 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 02:40:10.051332 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.051304 2576 fs.go:135] Filesystem UUIDs: map[347f77ed-0ba8-4798-baa0-e4bfd4b56ca6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cf248d36-83d7-4ada-8307-bc58a13917ed:/dev/nvme0n1p4] Jul 08 02:40:10.051436 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.051329 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 02:40:10.057368 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.057236 2576 manager.go:216] Machine: {Timestamp:2026-07-08 02:40:10.055971217 +0000 UTC m=+0.334348409 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099495 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f91c3df2144388cb1b2270eebfcd8 SystemUUID:ec2f91c3-df21-4438-8cb1-b2270eebfcd8 BootID:9610c092-ab09-4b56-8a80-a5420f0943a3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:82:43:a7:f4:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:82:43:a7:f4:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:25:dc:45:2c:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 02:40:10.057368 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.057339 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 02:40:10.057531 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.057441 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 02:40:10.060796 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.060766 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 02:40:10.060974 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.060799 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 02:40:10.061982 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.061969 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 02:40:10.062042 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.061986 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 02:40:10.062042 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.061996 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 02:40:10.062581 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.062568 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 02:40:10.062642 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.062597 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 02:40:10.063850 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.063837 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 02:40:10.066796 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.066783 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 02:40:10.066934 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.066923 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 02:40:10.066996 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.066960 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 08 02:40:10.066996 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.066979 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 02:40:10.067102 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.066997 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 02:40:10.067102 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.067047 2576 kubelet.go:398] "Adding apiserver pod source" Jul 08 02:40:10.067102 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.067068 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 02:40:10.067549 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.067504 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 02:40:10.067620 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.067560 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 02:40:10.071820 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.071606 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 02:40:10.073924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.073909 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 02:40:10.074000 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.073939 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 02:40:10.075336 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075323 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075340 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075347 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075368 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075378 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075384 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075392 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 02:40:10.075396 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075397 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 02:40:10.075576 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075403 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 02:40:10.075576 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075408 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 02:40:10.075576 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075416 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 02:40:10.075576 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.075421 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 02:40:10.076329 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.076320 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 02:40:10.076329 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.076328 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 02:40:10.079287 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.079274 2576 server.go:1268] "Started kubelet" Jul 08 02:40:10.079416 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.079385 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 02:40:10.079514 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.079460 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 02:40:10.080026 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.079964 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 02:40:10.080067 ip-10-0-128-107 systemd[1]: Started Kubernetes Kubelet. Jul 08 02:40:10.080214 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.080197 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 02:40:10.080271 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.080231 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 02:40:10.080321 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.080269 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 02:40:10.081140 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.081123 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 02:40:10.082652 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.082638 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 08 02:40:10.086244 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.086211 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 02:40:10.087054 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087039 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 02:40:10.087126 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087107 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 02:40:10.087177 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.087129 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.087659 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087637 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 02:40:10.087741 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087691 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 02:40:10.087803 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087636 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 02:40:10.088020 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087818 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 08 02:40:10.088020 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.087823 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 08 02:40:10.088110 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.087047 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-107.ec2.internal.18c0302a295a0dd4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-107.ec2.internal,UID:ip-10-0-128-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-107.ec2.internal,},FirstTimestamp:2026-07-08 02:40:10.079243732 +0000 UTC m=+0.357620927,LastTimestamp:2026-07-08 02:40:10.079243732 +0000 UTC m=+0.357620927,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-107.ec2.internal,}" Jul 08 02:40:10.088110 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088069 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 02:40:10.088110 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088082 2576 factory.go:55] Registering systemd factory Jul 08 02:40:10.088110 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088089 2576 factory.go:223] Registration of the systemd container factory successfully Jul 08 02:40:10.088418 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088278 2576 factory.go:153] Registering CRI-O factory Jul 08 02:40:10.088418 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088287 2576 factory.go:223] Registration of the crio container factory successfully Jul 08 02:40:10.088418 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088305 2576 factory.go:103] Registering Raw factory Jul 08 02:40:10.088418 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088317 2576 manager.go:1196] Started watching for new ooms in manager Jul 08 02:40:10.088634 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.088618 2576 manager.go:349] Starting recovery of all containers Jul 08 02:40:10.094619 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.094590 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 02:40:10.094746 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.094723 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 02:40:10.097001 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.096848 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dsxrc" Jul 08 02:40:10.099190 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.099167 2576 manager.go:354] Recovery completed Jul 08 02:40:10.100512 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.100458 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 02:40:10.103653 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.103634 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dsxrc" Jul 08 02:40:10.104622 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.104605 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.106545 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.106528 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.106595 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.106563 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.106595 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.106577 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.107079 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107057 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 02:40:10.107079 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107079 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 02:40:10.107177 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107095 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 02:40:10.107177 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107106 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.107791 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107776 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.107848 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107799 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.107848 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.107808 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.108127 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.108116 2576 policy_none.go:49] "None policy: Start" Jul 08 02:40:10.108190 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.108131 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 02:40:10.108190 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.108141 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 02:40:10.108961 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.108898 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-107.ec2.internal.18c0302a2afaa499 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-107.ec2.internal,UID:ip-10-0-128-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-107.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-107.ec2.internal,},FirstTimestamp:2026-07-08 02:40:10.106545305 +0000 UTC m=+0.384922504,LastTimestamp:2026-07-08 02:40:10.106545305 +0000 UTC m=+0.384922504,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-107.ec2.internal,}" Jul 08 02:40:10.109260 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.109250 2576 policy_none.go:47] "Start" Jul 08 02:40:10.142907 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.142887 2576 manager.go:337] "Starting Device Plugin manager" Jul 08 02:40:10.142991 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.142914 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 02:40:10.142991 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.142923 2576 server.go:87] "Starting device plugin registration server" Jul 08 02:40:10.143170 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.143157 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 02:40:10.143227 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.143178 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 02:40:10.143271 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.143261 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 02:40:10.143406 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.143394 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 02:40:10.143406 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.143407 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 02:40:10.143789 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.143769 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 02:40:10.143855 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.143820 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.243804 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.243788 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.244956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.244541 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.244956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.244561 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.244956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.244571 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.244956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.244597 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.254425 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.254399 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.254425 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.254420 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-107.ec2.internal\": node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.269422 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.269371 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.271792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.271777 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 02:40:10.271843 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.271811 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 02:40:10.271843 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.271827 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 02:40:10.271911 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.271853 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 02:40:10.274600 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.274583 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:40:10.370323 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.370302 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.372448 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.372431 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal"] Jul 08 02:40:10.372499 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.372483 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.374245 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.374230 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.374317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.374254 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.374317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.374263 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.375560 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.375548 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.375671 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.375650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.375709 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.375677 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.376179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.376164 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.376263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.376188 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.376263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.376199 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.376263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.376231 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.376263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.376255 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.376408 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.376267 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.377324 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.377304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.377324 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.377326 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:40:10.378787 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.378225 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:40:10.378787 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.378253 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:40:10.378787 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.378266 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:40:10.390645 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.390627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f42daac05cfc00507a8ce4234e1a889f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-107.ec2.internal\" (UID: \"f42daac05cfc00507a8ce4234e1a889f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.390714 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.390650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52b2295a2eb707313efa900307f9ea7b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal\" (UID: \"52b2295a2eb707313efa900307f9ea7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.390714 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.390668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b2295a2eb707313efa900307f9ea7b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal\" (UID: \"52b2295a2eb707313efa900307f9ea7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.394868 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.394851 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-107.ec2.internal\" not found" node="ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.398746 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.398733 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-107.ec2.internal\" not found" node="ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.470635 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.470616 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.491156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.491133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f42daac05cfc00507a8ce4234e1a889f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-107.ec2.internal\" (UID: \"f42daac05cfc00507a8ce4234e1a889f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.491156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.491149 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f42daac05cfc00507a8ce4234e1a889f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-107.ec2.internal\" (UID: \"f42daac05cfc00507a8ce4234e1a889f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.491236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.491174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52b2295a2eb707313efa900307f9ea7b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal\" (UID: \"52b2295a2eb707313efa900307f9ea7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.491236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.491199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b2295a2eb707313efa900307f9ea7b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal\" (UID: \"52b2295a2eb707313efa900307f9ea7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.491236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.491219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b2295a2eb707313efa900307f9ea7b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal\" (UID: \"52b2295a2eb707313efa900307f9ea7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.491326 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.491241 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52b2295a2eb707313efa900307f9ea7b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal\" (UID: \"52b2295a2eb707313efa900307f9ea7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.571668 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.571625 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.672107 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.672083 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.698547 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.698531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.702455 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:10.702383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:10.772330 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.772304 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.872869 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.872823 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:10.973316 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:10.973283 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:11.031007 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.030980 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 02:40:11.031569 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.031136 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 02:40:11.073599 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:11.073573 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:11.106459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.106418 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 02:35:10 +0000 UTC" deadline="2028-02-15 06:10:52.897662389 +0000 UTC" Jul 08 02:40:11.106459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.106446 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14091h30m41.791219875s" Jul 08 02:40:11.175449 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:11.175385 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:11.277601 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:11.277578 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:11.320010 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.319984 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:40:11.377661 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:11.377637 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-107.ec2.internal\" not found" Jul 08 02:40:11.439497 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.439445 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:40:11.487497 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.487477 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" Jul 08 02:40:11.499948 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.499931 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 02:40:11.501041 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.501027 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" Jul 08 02:40:11.510943 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.510925 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 02:40:11.568982 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:11.568956 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42daac05cfc00507a8ce4234e1a889f.slice/crio-17a0e70558e5273737cc8ffe17632f7a96deed2998df4075fb1c7184fb1bd384 WatchSource:0}: Error finding container 17a0e70558e5273737cc8ffe17632f7a96deed2998df4075fb1c7184fb1bd384: Status 404 returned error can't find the container with id 17a0e70558e5273737cc8ffe17632f7a96deed2998df4075fb1c7184fb1bd384 Jul 08 02:40:11.569653 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:11.569639 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b2295a2eb707313efa900307f9ea7b.slice/crio-cc1b18beb77259bd4b865b56d69523c4796a35e1520b0e3f5ced3eff3d941682 WatchSource:0}: Error finding container cc1b18beb77259bd4b865b56d69523c4796a35e1520b0e3f5ced3eff3d941682: Status 404 returned error can't find the container with id cc1b18beb77259bd4b865b56d69523c4796a35e1520b0e3f5ced3eff3d941682 Jul 08 02:40:11.575541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.575523 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:40:11.589494 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:11.589477 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:40:12.069909 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.069870 2576 apiserver.go:52] "Watching apiserver" Jul 08 02:40:12.079861 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.079831 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 02:40:12.080207 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.080183 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-84fsp","openshift-multus/multus-cgc8c","openshift-multus/network-metrics-daemon-frgkz","openshift-network-operator/iptables-alerter-xqjvf","openshift-ovn-kubernetes/ovnkube-node-rqdm8","kube-system/global-pull-secret-syncer-w26sd","kube-system/konnectivity-agent-5sdrk","kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qhr7s","openshift-dns/node-resolver-l5s2w","openshift-image-registry/node-ca-vfjph","openshift-network-diagnostics/network-check-target-kndhc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal"] Jul 08 02:40:12.082723 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.082705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.083980 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.083961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.085375 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.085343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.085535 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.085400 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.085535 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.085413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tvnb4\"" Jul 08 02:40:12.085535 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.085436 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:12.085535 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.085522 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 02:40:12.085767 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.085562 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.086280 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.086261 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.086817 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.086796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.087139 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.086932 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 02:40:12.087139 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.087025 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 02:40:12.087139 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.086934 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.087295 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.087268 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 02:40:12.087363 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.087350 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sgdzn\"" Jul 08 02:40:12.088288 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.088273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.089191 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.089171 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pmc7j\"" Jul 08 02:40:12.089462 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.089443 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 02:40:12.089583 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.089565 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 02:40:12.089891 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.089873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.090407 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.090378 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zf94f\"" Jul 08 02:40:12.090951 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.090931 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 02:40:12.091073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.090967 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 02:40:12.091073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.090960 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 02:40:12.091073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.090991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.091073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.090999 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.091475 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.091288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 02:40:12.091547 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.091526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.092064 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.092046 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 02:40:12.092064 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.092054 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.092686 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.092666 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.092778 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.092701 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7pwjq\"" Jul 08 02:40:12.093001 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.092984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:12.093067 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.093036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:12.094459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.094421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.095605 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.095587 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 02:40:12.095704 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.095634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 02:40:12.095762 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.095715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4qtjm\"" Jul 08 02:40:12.095898 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.095876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.096858 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.096843 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvbkw\"" Jul 08 02:40:12.096971 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.096953 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.097211 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.097194 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.097474 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.097449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.098381 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.098366 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.098778 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.098765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.099026 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.099004 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 02:40:12.099237 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.099222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x9l6g\"" Jul 08 02:40:12.099560 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.099540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.099621 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.099603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:12.100080 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 02:40:12.100162 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100112 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 02:40:12.100412 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67762\" (UniqueName: \"kubernetes.io/projected/010a311a-d396-497b-9742-78f12c75dfd8-kube-api-access-67762\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.100480 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-cni-bin\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100529 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-cni-multus\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100529 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100512 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8658c\"" Jul 08 02:40:12.100612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-hostroot\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66ct\" (UniqueName: \"kubernetes.io/projected/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-kube-api-access-d66ct\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.100767 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-cnibin\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.100767 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-system-cni-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100767 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-cni-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100767 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-cni-binary-copy\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-netns\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.100924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.100924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-run-netns\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.100924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-var-lib-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.100924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-etc-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.100963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-os-release\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-daemon-config\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-multus-certs\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-cni-netd\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-env-overrides\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-cnibin\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-socket-dir-parent\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101171 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-k8s-cni-cncf-io\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-kubelet\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmq8\" (UniqueName: \"kubernetes.io/projected/4e669e30-a095-4fe7-be16-7ca17010a100-kube-api-access-bqmq8\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ddr\" (UniqueName: \"kubernetes.io/projected/7df7babb-e20c-4d79-8b15-6e7c34341630-kube-api-access-s2ddr\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-systemd\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-ovn\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-conf-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-os-release\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78bede40-c8a7-4283-84bf-a036fcceda37-ovn-node-metrics-cert\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-slash\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.101590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-ovnkube-config\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-ovnkube-script-lib\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f56j\" (UniqueName: \"kubernetes.io/projected/78bede40-c8a7-4283-84bf-a036fcceda37-kube-api-access-5f56j\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-cni-binary-copy\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-kubelet\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-node-log\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-log-socket\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/010a311a-d396-497b-9742-78f12c75dfd8-host\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-etc-kubernetes\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-system-cni-dir\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.102107 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.101941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-systemd-units\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102658 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.102341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-cni-bin\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.102658 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.102473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/010a311a-d396-497b-9742-78f12c75dfd8-serviceca\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.103922 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.103898 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 02:40:12.126652 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.126591 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zgkn" Jul 08 02:40:12.139029 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.139007 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zgkn" Jul 08 02:40:12.190007 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.189976 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 02:40:12.203023 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.202996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-kubernetes\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.203162 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e920314d-549a-4cd1-8160-4decdd56ad45-kubelet-config\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.203162 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-daemon-config\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.203162 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-env-overrides\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203331 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69de278a-b53e-46db-88ce-bd314cd744f6-agent-certs\") pod \"konnectivity-agent-5sdrk\" (UID: \"69de278a-b53e-46db-88ce-bd314cd744f6\") " pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.203331 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-k8s-cni-cncf-io\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.203461 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-ovn\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203461 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjp9j\" (UniqueName: \"kubernetes.io/projected/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-kube-api-access-wjp9j\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.203461 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-conf-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.203461 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203428 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-k8s-cni-cncf-io\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-ovn\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-var-lib-kubelet\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-registration-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-slash\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-ovnkube-config\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-ovnkube-script-lib\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-env-overrides\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f56j\" (UniqueName: \"kubernetes.io/projected/78bede40-c8a7-4283-84bf-a036fcceda37-kube-api-access-5f56j\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysctl-d\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.203706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-slash\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.203806 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-daemon-config\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.204233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204077 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-ovnkube-config\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-conf-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.204233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e920314d-549a-4cd1-8160-4decdd56ad45-dbus\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.204233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-cni-binary-copy\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204234 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78bede40-c8a7-4283-84bf-a036fcceda37-ovnkube-script-lib\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-kubelet\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysconfig\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-device-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/010a311a-d396-497b-9742-78f12c75dfd8-host\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-systemd\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzkw\" (UniqueName: \"kubernetes.io/projected/576003a6-8598-44b6-a691-1828b97dd560-kube-api-access-2hzkw\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-host-slash\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:12.204414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/010a311a-d396-497b-9742-78f12c75dfd8-serviceca\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67762\" (UniqueName: \"kubernetes.io/projected/010a311a-d396-497b-9742-78f12c75dfd8-kube-api-access-67762\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-cni-multus\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204517 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/010a311a-d396-497b-9742-78f12c75dfd8-host\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66ct\" (UniqueName: \"kubernetes.io/projected/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-kube-api-access-d66ct\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-cni-binary-copy\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-node-log\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-cni-multus\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69de278a-b53e-46db-88ce-bd314cd744f6-konnectivity-ca\") pod \"konnectivity-agent-5sdrk\" (UID: \"69de278a-b53e-46db-88ce-bd314cd744f6\") " pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-kubelet\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-cni-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204746 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-node-log\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204802 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.204832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-var-lib-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204852 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-cni-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204936 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-var-lib-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbsl\" (UniqueName: \"kubernetes.io/projected/ba220b2f-2b74-4163-a98f-9975b0a172f5-kube-api-access-gvbsl\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.204976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-os-release\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-multus-certs\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/576003a6-8598-44b6-a691-1828b97dd560-tmp\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-multus-certs\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba220b2f-2b74-4163-a98f-9975b0a172f5-tmp-dir\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205143 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/010a311a-d396-497b-9742-78f12c75dfd8-serviceca\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205193 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-os-release\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddzt\" (UniqueName: \"kubernetes.io/projected/59e3da85-4326-4bad-8b4e-68b890e5ce1d-kube-api-access-cddzt\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-cnibin\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-socket-dir-parent\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-kubelet\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.205606 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmq8\" (UniqueName: \"kubernetes.io/projected/4e669e30-a095-4fe7-be16-7ca17010a100-kube-api-access-bqmq8\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-multus-socket-dir-parent\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-kubelet\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205469 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-cnibin\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ddr\" (UniqueName: \"kubernetes.io/projected/7df7babb-e20c-4d79-8b15-6e7c34341630-kube-api-access-s2ddr\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-systemd\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78bede40-c8a7-4283-84bf-a036fcceda37-ovn-node-metrics-cert\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-os-release\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysctl-conf\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205673 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-run-systemd\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-lib-modules\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba220b2f-2b74-4163-a98f-9975b0a172f5-hosts-file\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-os-release\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-sys-fs\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.206449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-log-socket\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-sys\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205919 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-log-socket\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/576003a6-8598-44b6-a691-1828b97dd560-etc-tuned\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-iptables-alerter-script\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-etc-kubernetes\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.205996 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-etc-kubernetes\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-system-cni-dir\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-systemd-units\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206068 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-system-cni-dir\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-cni-bin\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206110 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-systemd-units\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-run\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7df7babb-e20c-4d79-8b15-6e7c34341630-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-host\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-cni-bin\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.207298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-socket-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-cni-bin\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-hostroot\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206299 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-var-lib-cni-bin\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-cnibin\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7df7babb-e20c-4d79-8b15-6e7c34341630-cnibin\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-run-netns\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.206483 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-etc-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-run-netns\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-hostroot\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.206558 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:12.706536616 +0000 UTC m=+2.984913801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206576 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-etc-openvswitch\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206586 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.208179 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-system-cni-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-cni-binary-copy\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206726 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-system-cni-dir\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-netns\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-cni-netd\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-modprobe-d\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-host-run-netns\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.206866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78bede40-c8a7-4283-84bf-a036fcceda37-host-cni-netd\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.209099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.207189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-cni-binary-copy\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.210570 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.210546 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78bede40-c8a7-4283-84bf-a036fcceda37-ovn-node-metrics-cert\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.213629 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.213603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66ct\" (UniqueName: \"kubernetes.io/projected/1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20-kube-api-access-d66ct\") pod \"multus-cgc8c\" (UID: \"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20\") " pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.213803 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.213784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f56j\" (UniqueName: \"kubernetes.io/projected/78bede40-c8a7-4283-84bf-a036fcceda37-kube-api-access-5f56j\") pod \"ovnkube-node-rqdm8\" (UID: \"78bede40-c8a7-4283-84bf-a036fcceda37\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.215792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.215762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67762\" (UniqueName: \"kubernetes.io/projected/010a311a-d396-497b-9742-78f12c75dfd8-kube-api-access-67762\") pod \"node-ca-vfjph\" (UID: \"010a311a-d396-497b-9742-78f12c75dfd8\") " pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.219241 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.219219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ddr\" (UniqueName: \"kubernetes.io/projected/7df7babb-e20c-4d79-8b15-6e7c34341630-kube-api-access-s2ddr\") pod \"multus-additional-cni-plugins-84fsp\" (UID: \"7df7babb-e20c-4d79-8b15-6e7c34341630\") " pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.219625 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.219608 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmq8\" (UniqueName: \"kubernetes.io/projected/4e669e30-a095-4fe7-be16-7ca17010a100-kube-api-access-bqmq8\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.282203 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.282151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" event={"ID":"52b2295a2eb707313efa900307f9ea7b","Type":"ContainerStarted","Data":"cc1b18beb77259bd4b865b56d69523c4796a35e1520b0e3f5ced3eff3d941682"} Jul 08 02:40:12.283485 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.283456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" event={"ID":"f42daac05cfc00507a8ce4234e1a889f","Type":"ContainerStarted","Data":"17a0e70558e5273737cc8ffe17632f7a96deed2998df4075fb1c7184fb1bd384"} Jul 08 02:40:12.307548 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-systemd\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.307656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzkw\" (UniqueName: \"kubernetes.io/projected/576003a6-8598-44b6-a691-1828b97dd560-kube-api-access-2hzkw\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.307656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-host-slash\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.307656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:12.307656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69de278a-b53e-46db-88ce-bd314cd744f6-konnectivity-ca\") pod \"konnectivity-agent-5sdrk\" (UID: \"69de278a-b53e-46db-88ce-bd314cd744f6\") " pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbsl\" (UniqueName: \"kubernetes.io/projected/ba220b2f-2b74-4163-a98f-9975b0a172f5-kube-api-access-gvbsl\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-host-slash\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/576003a6-8598-44b6-a691-1828b97dd560-tmp\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba220b2f-2b74-4163-a98f-9975b0a172f5-tmp-dir\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddzt\" (UniqueName: \"kubernetes.io/projected/59e3da85-4326-4bad-8b4e-68b890e5ce1d-kube-api-access-cddzt\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysctl-conf\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.307883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-lib-modules\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba220b2f-2b74-4163-a98f-9975b0a172f5-hosts-file\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-sys-fs\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-sys\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.307968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/576003a6-8598-44b6-a691-1828b97dd560-etc-tuned\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-iptables-alerter-script\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-run\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69de278a-b53e-46db-88ce-bd314cd744f6-konnectivity-ca\") pod \"konnectivity-agent-5sdrk\" (UID: \"69de278a-b53e-46db-88ce-bd314cd744f6\") " pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-host\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-socket-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-modprobe-d\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-kubernetes\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e920314d-549a-4cd1-8160-4decdd56ad45-kubelet-config\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.308400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-modprobe-d\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-socket-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69de278a-b53e-46db-88ce-bd314cd744f6-agent-certs\") pod \"konnectivity-agent-5sdrk\" (UID: \"69de278a-b53e-46db-88ce-bd314cd744f6\") " pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308484 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-kubernetes\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjp9j\" (UniqueName: \"kubernetes.io/projected/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-kube-api-access-wjp9j\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-var-lib-kubelet\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-registration-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-iptables-alerter-script\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysctl-d\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e920314d-549a-4cd1-8160-4decdd56ad45-dbus\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysconfig\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-device-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysctl-d\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308745 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysconfig\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-device-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-lib-modules\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308787 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-host\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308800 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-var-lib-kubelet\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308830 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba220b2f-2b74-4163-a98f-9975b0a172f5-hosts-file\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-registration-dir\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59e3da85-4326-4bad-8b4e-68b890e5ce1d-sys-fs\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308912 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e920314d-549a-4cd1-8160-4decdd56ad45-dbus\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.308975 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308979 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-sys\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.308440 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-systemd\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.309046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:12.809024632 +0000 UTC m=+3.087401814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.309105 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-etc-sysctl-conf\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.309122 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba220b2f-2b74-4163-a98f-9975b0a172f5-tmp-dir\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.309152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e920314d-549a-4cd1-8160-4decdd56ad45-kubelet-config\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.309945 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.309218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/576003a6-8598-44b6-a691-1828b97dd560-run\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.310681 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.310623 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/576003a6-8598-44b6-a691-1828b97dd560-etc-tuned\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.310891 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.310875 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/576003a6-8598-44b6-a691-1828b97dd560-tmp\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.311695 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.311671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69de278a-b53e-46db-88ce-bd314cd744f6-agent-certs\") pod \"konnectivity-agent-5sdrk\" (UID: \"69de278a-b53e-46db-88ce-bd314cd744f6\") " pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.330554 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.330495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzkw\" (UniqueName: \"kubernetes.io/projected/576003a6-8598-44b6-a691-1828b97dd560-kube-api-access-2hzkw\") pod \"tuned-qhr7s\" (UID: \"576003a6-8598-44b6-a691-1828b97dd560\") " pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.330684 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.330662 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:12.330821 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.330690 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:12.330821 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.330706 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:12.330821 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.330776 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:40:12.830758757 +0000 UTC m=+3.109135955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:12.333239 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.333215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjp9j\" (UniqueName: \"kubernetes.io/projected/896d89d5-5909-4a09-b6a5-8ff33dd6eaba-kube-api-access-wjp9j\") pod \"iptables-alerter-xqjvf\" (UID: \"896d89d5-5909-4a09-b6a5-8ff33dd6eaba\") " pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.333699 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.333676 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbsl\" (UniqueName: \"kubernetes.io/projected/ba220b2f-2b74-4163-a98f-9975b0a172f5-kube-api-access-gvbsl\") pod \"node-resolver-l5s2w\" (UID: \"ba220b2f-2b74-4163-a98f-9975b0a172f5\") " pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.334113 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.334097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddzt\" (UniqueName: \"kubernetes.io/projected/59e3da85-4326-4bad-8b4e-68b890e5ce1d-kube-api-access-cddzt\") pod \"aws-ebs-csi-driver-node-z9khc\" (UID: \"59e3da85-4326-4bad-8b4e-68b890e5ce1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.394979 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.394951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vfjph" Jul 08 02:40:12.402380 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.402356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cgc8c" Jul 08 02:40:12.403487 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.403462 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010a311a_d396_497b_9742_78f12c75dfd8.slice/crio-7c913049133a172e5c11ea51b7e60bcc7b8f663120d64a22c523de5a308a3df0 WatchSource:0}: Error finding container 7c913049133a172e5c11ea51b7e60bcc7b8f663120d64a22c523de5a308a3df0: Status 404 returned error can't find the container with id 7c913049133a172e5c11ea51b7e60bcc7b8f663120d64a22c523de5a308a3df0 Jul 08 02:40:12.410333 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.410317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84fsp" Jul 08 02:40:12.412929 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.412898 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0f6a7b_d7fb_4a5e_a7a6_9704dc9b7d20.slice/crio-2d40b7c019d19c58c7f588c341d1c1443039c034eeb350642739ce9f2dbcd22c WatchSource:0}: Error finding container 2d40b7c019d19c58c7f588c341d1c1443039c034eeb350642739ce9f2dbcd22c: Status 404 returned error can't find the container with id 2d40b7c019d19c58c7f588c341d1c1443039c034eeb350642739ce9f2dbcd22c Jul 08 02:40:12.415798 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.415778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:12.421953 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.421925 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df7babb_e20c_4d79_8b15_6e7c34341630.slice/crio-362dbd4d2c470800a8fff454277dfe501dbfa8e13d2131373eec5d621ed319c7 WatchSource:0}: Error finding container 362dbd4d2c470800a8fff454277dfe501dbfa8e13d2131373eec5d621ed319c7: Status 404 returned error can't find the container with id 362dbd4d2c470800a8fff454277dfe501dbfa8e13d2131373eec5d621ed319c7 Jul 08 02:40:12.424609 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.424589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" Jul 08 02:40:12.427837 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.427808 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bede40_c8a7_4283_84bf_a036fcceda37.slice/crio-5ba33886490dc6dad86d4b04ff10233f74c79ded619abbbc80110ae6750bb515 WatchSource:0}: Error finding container 5ba33886490dc6dad86d4b04ff10233f74c79ded619abbbc80110ae6750bb515: Status 404 returned error can't find the container with id 5ba33886490dc6dad86d4b04ff10233f74c79ded619abbbc80110ae6750bb515 Jul 08 02:40:12.431284 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.431263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:12.436506 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.436472 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e3da85_4326_4bad_8b4e_68b890e5ce1d.slice/crio-31d42f0466fe774010cef2fd9dcf2a36d16e387965b76d96ce4500b345776630 WatchSource:0}: Error finding container 31d42f0466fe774010cef2fd9dcf2a36d16e387965b76d96ce4500b345776630: Status 404 returned error can't find the container with id 31d42f0466fe774010cef2fd9dcf2a36d16e387965b76d96ce4500b345776630 Jul 08 02:40:12.439235 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.438956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" Jul 08 02:40:12.441351 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.441328 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69de278a_b53e_46db_88ce_bd314cd744f6.slice/crio-ceaa451c24a307d77d7803f66181ab57815a51eadcc83b7c33a364622e88c26d WatchSource:0}: Error finding container ceaa451c24a307d77d7803f66181ab57815a51eadcc83b7c33a364622e88c26d: Status 404 returned error can't find the container with id ceaa451c24a307d77d7803f66181ab57815a51eadcc83b7c33a364622e88c26d Jul 08 02:40:12.448714 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.448443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqjvf" Jul 08 02:40:12.454052 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.454031 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5s2w" Jul 08 02:40:12.465720 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.465674 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576003a6_8598_44b6_a691_1828b97dd560.slice/crio-9abe8861154dd909fef0bbce3d419e0cf6d2da44db51cb56431554a815344a25 WatchSource:0}: Error finding container 9abe8861154dd909fef0bbce3d419e0cf6d2da44db51cb56431554a815344a25: Status 404 returned error can't find the container with id 9abe8861154dd909fef0bbce3d419e0cf6d2da44db51cb56431554a815344a25 Jul 08 02:40:12.471147 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:12.471120 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba220b2f_2b74_4163_a98f_9975b0a172f5.slice/crio-83b0c9f2769b92b8e105d0ec32075676165435f98671db4e398bc7f9a2380227 WatchSource:0}: Error finding container 83b0c9f2769b92b8e105d0ec32075676165435f98671db4e398bc7f9a2380227: Status 404 returned error can't find the container with id 83b0c9f2769b92b8e105d0ec32075676165435f98671db4e398bc7f9a2380227 Jul 08 02:40:12.593617 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.593539 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:40:12.712795 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.712759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:12.712981 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.712939 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:12.713046 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.713002 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:13.712983182 +0000 UTC m=+3.991360364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:12.813657 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.813613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:12.813856 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.813829 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:12.813924 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.813894 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:13.813872701 +0000 UTC m=+4.092249894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:12.914973 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:12.914934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:12.915645 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.915164 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:12.915645 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.915193 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:12.915645 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.915207 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:12.915645 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:12.915273 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:40:13.915252265 +0000 UTC m=+4.193629451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:13.141573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.141536 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 02:35:12 +0000 UTC" deadline="2027-12-23 19:15:20.035496271 +0000 UTC" Jul 08 02:40:13.141573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.141568 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12808h35m6.893932467s" Jul 08 02:40:13.288506 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.288402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqjvf" event={"ID":"896d89d5-5909-4a09-b6a5-8ff33dd6eaba","Type":"ContainerStarted","Data":"15e15a824e61d7b38dad4ae9ef9c729ae4eb18e790a9fb8afc4c0bf356abc620"} Jul 08 02:40:13.291237 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.291196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" event={"ID":"576003a6-8598-44b6-a691-1828b97dd560","Type":"ContainerStarted","Data":"9abe8861154dd909fef0bbce3d419e0cf6d2da44db51cb56431554a815344a25"} Jul 08 02:40:13.293357 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.293329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sdrk" event={"ID":"69de278a-b53e-46db-88ce-bd314cd744f6","Type":"ContainerStarted","Data":"ceaa451c24a307d77d7803f66181ab57815a51eadcc83b7c33a364622e88c26d"} Jul 08 02:40:13.295847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.295818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" event={"ID":"59e3da85-4326-4bad-8b4e-68b890e5ce1d","Type":"ContainerStarted","Data":"31d42f0466fe774010cef2fd9dcf2a36d16e387965b76d96ce4500b345776630"} Jul 08 02:40:13.298707 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.298676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerStarted","Data":"362dbd4d2c470800a8fff454277dfe501dbfa8e13d2131373eec5d621ed319c7"} Jul 08 02:40:13.300733 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.300701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vfjph" event={"ID":"010a311a-d396-497b-9742-78f12c75dfd8","Type":"ContainerStarted","Data":"7c913049133a172e5c11ea51b7e60bcc7b8f663120d64a22c523de5a308a3df0"} Jul 08 02:40:13.302138 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.302112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5s2w" event={"ID":"ba220b2f-2b74-4163-a98f-9975b0a172f5","Type":"ContainerStarted","Data":"83b0c9f2769b92b8e105d0ec32075676165435f98671db4e398bc7f9a2380227"} Jul 08 02:40:13.303912 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.303884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"5ba33886490dc6dad86d4b04ff10233f74c79ded619abbbc80110ae6750bb515"} Jul 08 02:40:13.306114 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.306090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cgc8c" event={"ID":"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20","Type":"ContainerStarted","Data":"2d40b7c019d19c58c7f588c341d1c1443039c034eeb350642739ce9f2dbcd22c"} Jul 08 02:40:13.721714 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.721671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:13.721916 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.721887 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:13.721982 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.721955 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:15.721938139 +0000 UTC m=+6.000315342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:13.823063 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.823021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:13.823280 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.823255 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:13.823370 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.823330 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:15.823309282 +0000 UTC m=+6.101686467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:13.924199 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:13.924157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:13.924372 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.924332 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:13.924372 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.924355 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:13.924372 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.924368 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:13.924578 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:13.924472 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:40:15.92444862 +0000 UTC m=+6.202825804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:14.142245 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:14.142205 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 02:35:12 +0000 UTC" deadline="2028-01-07 14:10:02.445206469 +0000 UTC" Jul 08 02:40:14.142245 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:14.142245 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13163h29m48.302966404s" Jul 08 02:40:14.273992 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:14.273543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:14.274155 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:14.274111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:14.274155 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:14.273970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:14.274280 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:14.274223 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:14.274280 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:14.273949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:14.274541 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:14.274407 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:14.318549 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:14.318504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" event={"ID":"f42daac05cfc00507a8ce4234e1a889f","Type":"ContainerStarted","Data":"0638153fa340554d61a3b5222738c9b50a8c15a8f749ec5b70519c2cc79222b2"} Jul 08 02:40:15.351905 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:15.351811 2576 generic.go:356] "Generic (PLEG): container finished" podID="52b2295a2eb707313efa900307f9ea7b" containerID="98a05ca0b2e4289a62228147c05cd82eb01572f167f65a636689b30d4310a04f" exitCode=0 Jul 08 02:40:15.352908 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:15.352867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" event={"ID":"52b2295a2eb707313efa900307f9ea7b","Type":"ContainerDied","Data":"98a05ca0b2e4289a62228147c05cd82eb01572f167f65a636689b30d4310a04f"} Jul 08 02:40:15.369328 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:15.369270 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-107.ec2.internal" podStartSLOduration=3.369247696 podStartE2EDuration="3.369247696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:14.334066315 +0000 UTC m=+4.612443522" watchObservedRunningTime="2026-07-08 02:40:15.369247696 +0000 UTC m=+5.647624903" Jul 08 02:40:15.745200 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:15.744525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:15.745200 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.744754 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:15.745200 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.744819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:19.744799313 +0000 UTC m=+10.023176510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:15.845923 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:15.845334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:15.845923 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.845524 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:15.845923 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.845581 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:19.845563369 +0000 UTC m=+10.123940555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:15.947020 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:15.946289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:15.947020 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.946483 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:15.947020 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.946508 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:15.947020 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.946522 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:15.947020 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:15.946583 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:40:19.946564105 +0000 UTC m=+10.224941290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:16.275098 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:16.275004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:16.275274 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:16.275154 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:16.275274 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:16.275265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:16.275391 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:16.275364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:16.275471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:16.275412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:16.275540 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:16.275499 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:16.358329 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:16.358286 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" event={"ID":"52b2295a2eb707313efa900307f9ea7b","Type":"ContainerStarted","Data":"6292f880030a4ac62abf802a13799cd130cf3b7ca53e15c254e75203e7ef7799"} Jul 08 02:40:18.273402 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:18.272579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:18.273402 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:18.272735 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:18.273402 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:18.273208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:18.273402 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:18.273307 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:18.274038 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:18.273525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:18.274038 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:18.273643 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:19.780120 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:19.780079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:19.780600 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.780239 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:19.780600 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.780381 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:27.780340086 +0000 UTC m=+18.058717269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:19.881488 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:19.881449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:19.881665 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.881642 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:19.881798 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.881715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:27.881694884 +0000 UTC m=+18.160072083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:19.982740 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:19.982701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:19.982914 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.982897 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:19.982970 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.982922 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:19.982970 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.982937 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:19.983060 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:19.983000 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:40:27.982980617 +0000 UTC m=+18.261357801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:20.274230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:20.273551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:20.274230 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:20.273647 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:20.274230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:20.273962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:20.274230 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:20.274048 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:20.274230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:20.274087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:20.274230 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:20.274151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:22.275479 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:22.274701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:22.275479 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:22.274831 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:22.275479 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:22.274905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:22.275479 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:22.274985 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:22.275479 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:22.275033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:22.275479 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:22.275093 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:24.273447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:24.272747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:24.273447 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:24.272884 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:24.273447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:24.273253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:24.273447 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:24.273342 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:24.273915 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:24.273479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:24.273915 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:24.273564 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:26.272447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:26.272413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:26.272897 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:26.272461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:26.272897 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:26.272423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:26.272897 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:26.272541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:26.272897 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:26.272615 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:26.272897 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:26.272699 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:27.844736 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:27.844694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:27.845176 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:27.844869 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:27.845176 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:27.844934 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:43.84491866 +0000 UTC m=+34.123295839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:27.945175 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:27.945142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:27.945443 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:27.945247 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:27.945443 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:27.945297 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:43.945284315 +0000 UTC m=+34.223661495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:28.046120 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:28.046084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:28.046277 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.046226 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:28.046277 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.046246 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:28.046277 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.046258 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:28.046470 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.046322 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:40:44.046301389 +0000 UTC m=+34.324678569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:28.272995 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:28.272913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:28.272995 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:28.272955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:28.273191 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:28.272922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:28.273191 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.273053 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:28.273191 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.273140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:28.273329 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:28.273208 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:30.273278 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:30.273247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:30.273688 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:30.273370 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:30.273688 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:30.273374 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:30.273688 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:30.273506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:30.273688 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:30.273535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:30.273688 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:30.273593 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:32.272706 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:32.272674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:32.273245 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:32.272674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:32.273245 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:32.272803 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:32.273245 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:32.272673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:32.273245 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:32.272875 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:32.273245 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:32.272950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:33.393387 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.393029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" event={"ID":"576003a6-8598-44b6-a691-1828b97dd560","Type":"ContainerStarted","Data":"ee9dda47bf9fe7653eeddee2c627efe220bdbb43a4aca6850ae9dbe4e64a6c87"} Jul 08 02:40:33.394597 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.394572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sdrk" event={"ID":"69de278a-b53e-46db-88ce-bd314cd744f6","Type":"ContainerStarted","Data":"9b02bffac05e1ae4088c9d2441cad85dd5a252288e57ee12da650d4bd364402d"} Jul 08 02:40:33.395875 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.395847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" event={"ID":"59e3da85-4326-4bad-8b4e-68b890e5ce1d","Type":"ContainerStarted","Data":"5442e66740034f1d587f6359f6292ab7008a558edbc46ec20dfae109394589c5"} Jul 08 02:40:33.397153 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.397130 2576 generic.go:356] "Generic (PLEG): container finished" podID="7df7babb-e20c-4d79-8b15-6e7c34341630" containerID="395bb3c0521f3c86b39c7a7b7e4480ecfcb5a591a96596d1b73a0e8efe8fd54e" exitCode=0 Jul 08 02:40:33.397242 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.397214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerDied","Data":"395bb3c0521f3c86b39c7a7b7e4480ecfcb5a591a96596d1b73a0e8efe8fd54e"} Jul 08 02:40:33.398751 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.398722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vfjph" event={"ID":"010a311a-d396-497b-9742-78f12c75dfd8","Type":"ContainerStarted","Data":"4d89d497971512fb003dcf7018a807d8e3294296b832d6c04b5ca394d58f8793"} Jul 08 02:40:33.400305 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.400110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5s2w" event={"ID":"ba220b2f-2b74-4163-a98f-9975b0a172f5","Type":"ContainerStarted","Data":"71e81a10eb7c18b8842d58bfa0f9be4d9c9e1639b47905e261be300b1364d6b7"} Jul 08 02:40:33.403085 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.403066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"4c90982df2924eba8b5586ac806f805782a5a0b42cdc5bcf0a7bc139f9c3d767"} Jul 08 02:40:33.403165 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.403130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"64e817e59f20326fd730f47fe877234e4c03f468a3180fa7ae499744642c699a"} Jul 08 02:40:33.403211 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.403171 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"28af3fe27b71d694f5b5d3f469e710c2922f419d26a511377a0f2143df58dac4"} Jul 08 02:40:33.403211 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.403207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"803985dd18fef5dc0895f64184d1e566d380b0410b9eb00c300fe6b1d493e633"} Jul 08 02:40:33.403282 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.403239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"bc6f66275c9c9aa0dd486b0599e80e5a64610823a75c66f7581d914e2c485de4"} Jul 08 02:40:33.403282 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.403275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"f9fb66a62fa2ab3d6a156827ed8291b6fa0c47421587470897c64519a8c82a68"} Jul 08 02:40:33.404794 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.404776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cgc8c" event={"ID":"1d0f6a7b-d7fb-4a5e-a7a6-9704dc9b7d20","Type":"ContainerStarted","Data":"b717a3fdfc158cb2c7ffcb054a99365f154f4b59d46ca470b3429ce61ce0a436"} Jul 08 02:40:33.429916 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.429867 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qhr7s" podStartSLOduration=3.406966418 podStartE2EDuration="23.42984026s" totalImagesPullingTime="20.022873842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:33.428947808 +0000 UTC m=+23.707325027" watchObservedRunningTime="2026-07-08 02:40:33.42984026 +0000 UTC m=+23.708217463" Jul 08 02:40:33.430117 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.430098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-107.ec2.internal" podStartSLOduration=21.430093665 podStartE2EDuration="21.430093665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:16.375958701 +0000 UTC m=+6.654335908" watchObservedRunningTime="2026-07-08 02:40:33.430093665 +0000 UTC m=+23.708470866" Jul 08 02:40:33.503777 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.503737 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5sdrk" podStartSLOduration=11.930847731 podStartE2EDuration="23.503725561s" totalImagesPullingTime="11.57287783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:33.455044999 +0000 UTC m=+23.733422214" watchObservedRunningTime="2026-07-08 02:40:33.503725561 +0000 UTC m=+23.782102760" Jul 08 02:40:33.536732 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.536691 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l5s2w" podStartSLOduration=10.994555599 podStartE2EDuration="22.536679095s" totalImagesPullingTime="11.542123496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:33.504344143 +0000 UTC m=+23.782721345" watchObservedRunningTime="2026-07-08 02:40:33.536679095 +0000 UTC m=+23.815056337" Jul 08 02:40:33.570490 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:33.570455 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cgc8c" podStartSLOduration=3.429623322 podStartE2EDuration="23.570442632s" totalImagesPullingTime="20.14081931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:33.536329439 +0000 UTC m=+23.814706632" watchObservedRunningTime="2026-07-08 02:40:33.570442632 +0000 UTC m=+23.848819833" Jul 08 02:40:34.272636 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:34.272604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:34.272952 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:34.272605 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:34.272952 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:34.272733 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:34.272952 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:34.272824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:34.272952 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:34.272886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:34.273157 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:34.272952 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:34.408773 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:34.408736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqjvf" event={"ID":"896d89d5-5909-4a09-b6a5-8ff33dd6eaba","Type":"ContainerStarted","Data":"7f151f8c539d164cdda1fa2f542bc94d9a40215b01e4eb487bb9e07777cb839e"} Jul 08 02:40:34.435568 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:34.435518 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vfjph" podStartSLOduration=12.823143968 podStartE2EDuration="24.435498034s" totalImagesPullingTime="11.612354066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:33.598226049 +0000 UTC m=+23.876603278" watchObservedRunningTime="2026-07-08 02:40:34.435498034 +0000 UTC m=+24.713875237" Jul 08 02:40:34.584303 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:34.584284 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 02:40:35.058429 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.058386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:35.058928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.058907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:35.146254 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.146207 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xqjvf" podStartSLOduration=12.590227658 podStartE2EDuration="24.146192581s" totalImagesPullingTime="11.555964923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:34.43504727 +0000 UTC m=+24.713424487" watchObservedRunningTime="2026-07-08 02:40:35.146192581 +0000 UTC m=+25.424569786" Jul 08 02:40:35.163862 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.163740 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T02:40:34.584299986Z","UUID":"ab3b70f5-b732-481f-a955-8a71952088d5","Handler":null,"Name":"","Endpoint":""} Jul 08 02:40:35.165196 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.165171 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 02:40:35.165302 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.165208 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 02:40:35.413481 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.413445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" event={"ID":"59e3da85-4326-4bad-8b4e-68b890e5ce1d","Type":"ContainerStarted","Data":"ce89afdf3b1e864d3395fc7972c334992af2da493a9aa6e26d2d04e485b87466"} Jul 08 02:40:35.417977 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.417947 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"aa5ddb78683a2229165d25611c512e009832c210f02d090275e8cd87b7c38027"} Jul 08 02:40:35.418227 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.418172 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:35.418590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.418571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5sdrk" Jul 08 02:40:35.991857 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:35.991824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:36.109725 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.109699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:36.276484 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.276242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:36.276656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.276252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:36.276656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.276275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:36.276656 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:36.276610 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:36.276802 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:36.276757 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:36.276856 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:36.276827 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:36.322438 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.322402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:36.423450 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.423415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" event={"ID":"59e3da85-4326-4bad-8b4e-68b890e5ce1d","Type":"ContainerStarted","Data":"2e55439183acc32b0b8702f1ae81475bc303bda1e4e5d881da7e20d08c42926e"} Jul 08 02:40:36.427835 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:36.427764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:37.036981 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:37.036946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:37.113906 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:37.113877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:37.430891 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:37.430844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:37.532524 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:37.532492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:38.272992 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.272764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:38.273147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.272771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:38.273147 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:38.273049 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:38.273258 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:38.273140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:38.273258 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.272801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:38.273258 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:38.273235 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:38.431418 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.431382 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" event={"ID":"78bede40-c8a7-4283-84bf-a036fcceda37","Type":"ContainerStarted","Data":"6f76070bd1b2dc2fd3f0d950c4012cadb8f416154ed4020571431ad001127220"} Jul 08 02:40:38.432276 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.431571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:38.432276 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.431628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:38.432276 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.431753 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:38.449966 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.449942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:38.450119 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.450067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:40:38.461234 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.461191 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9khc" podStartSLOduration=13.623411265 podStartE2EDuration="28.46117664s" totalImagesPullingTime="14.837765375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:36.470175762 +0000 UTC m=+26.748552984" watchObservedRunningTime="2026-07-08 02:40:38.46117664 +0000 UTC m=+28.739553867" Jul 08 02:40:38.461610 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.461581 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" podStartSLOduration=8.336125583 podStartE2EDuration="28.461567182s" totalImagesPullingTime="20.125441599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:38.460947954 +0000 UTC m=+28.739325152" watchObservedRunningTime="2026-07-08 02:40:38.461567182 +0000 UTC m=+28.739944386" Jul 08 02:40:38.828054 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:38.827919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:39.505822 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:39.505777 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-frgkz"] Jul 08 02:40:39.506258 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:39.505962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:39.506258 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:39.506080 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:39.509926 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:39.509898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w26sd"] Jul 08 02:40:39.510133 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:39.510049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:39.510406 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:39.510341 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:39.510552 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:39.510529 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kndhc"] Jul 08 02:40:39.510638 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:39.510625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:39.510708 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:39.510689 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:40.438378 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:40.438332 2576 generic.go:356] "Generic (PLEG): container finished" podID="7df7babb-e20c-4d79-8b15-6e7c34341630" containerID="8eabfbc527796848a020238906bdc6d7e3ff3e85f16252fd45aefec1395624ba" exitCode=0 Jul 08 02:40:40.438573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:40.438394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerDied","Data":"8eabfbc527796848a020238906bdc6d7e3ff3e85f16252fd45aefec1395624ba"} Jul 08 02:40:41.272161 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:41.272136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:41.272480 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:41.272169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:41.272480 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:41.272235 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:41.272480 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:41.272314 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:41.272480 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:41.272397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:41.272603 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:41.272504 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:42.445442 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:42.445190 2576 generic.go:356] "Generic (PLEG): container finished" podID="7df7babb-e20c-4d79-8b15-6e7c34341630" containerID="6afa728452c387bae065760df1e04fe5c251a64864d998bda0a0eb11e42aa6a1" exitCode=0 Jul 08 02:40:42.445442 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:42.445262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerDied","Data":"6afa728452c387bae065760df1e04fe5c251a64864d998bda0a0eb11e42aa6a1"} Jul 08 02:40:43.272673 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.272636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:43.272810 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.272635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:43.272810 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.272753 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:43.272888 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.272805 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:43.272888 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.272655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:43.272888 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.272863 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:43.294762 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.294737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:43.399928 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.399900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:43.450246 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.450215 2576 generic.go:356] "Generic (PLEG): container finished" podID="7df7babb-e20c-4d79-8b15-6e7c34341630" containerID="75793d340ad05fe73452fedc312693c002906d3ee518b36dc5dde70f9f8d0bc7" exitCode=0 Jul 08 02:40:43.450703 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.450272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerDied","Data":"75793d340ad05fe73452fedc312693c002906d3ee518b36dc5dde70f9f8d0bc7"} Jul 08 02:40:43.868540 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.868466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:43.868669 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.868630 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:43.868714 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.868694 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:41:15.868680233 +0000 UTC m=+66.147057418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:40:43.969658 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:43.969628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:43.969848 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.969754 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:43.969848 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:43.969809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret podName:e920314d-549a-4cd1-8160-4decdd56ad45 nodeName:}" failed. No retries permitted until 2026-07-08 02:41:15.969794985 +0000 UTC m=+66.248172165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret") pod "global-pull-secret-syncer-w26sd" (UID: "e920314d-549a-4cd1-8160-4decdd56ad45") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:40:44.070846 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:44.070813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:44.071023 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:44.070950 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:40:44.071023 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:44.070968 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:40:44.071023 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:44.070981 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4h68p for pod openshift-network-diagnostics/network-check-target-kndhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:44.071192 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:44.071042 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p podName:a82ebec2-db66-4718-b77e-22d95f88692b nodeName:}" failed. No retries permitted until 2026-07-08 02:41:16.071025934 +0000 UTC m=+66.349403137 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4h68p" (UniqueName: "kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p") pod "network-check-target-kndhc" (UID: "a82ebec2-db66-4718-b77e-22d95f88692b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:40:45.097070 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.097026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:45.179586 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.179554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:45.272608 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.272541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:45.272608 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.272556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:45.272608 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.272567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:45.272821 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:45.272661 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w26sd" podUID="e920314d-549a-4cd1-8160-4decdd56ad45" Jul 08 02:40:45.272821 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:45.272809 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:40:45.272905 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:45.272885 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kndhc" podUID="a82ebec2-db66-4718-b77e-22d95f88692b" Jul 08 02:40:45.435935 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.435908 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-107.ec2.internal" event="NodeReady" Jul 08 02:40:45.436086 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.436058 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 02:40:45.489374 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.489329 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tdxfs"] Jul 08 02:40:45.491507 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.491478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.494316 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.494285 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 02:40:45.494316 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.494306 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 02:40:45.494510 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.494425 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8d2rg\"" Jul 08 02:40:45.495539 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.495519 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8q9px"] Jul 08 02:40:45.497230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.497212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.499853 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.499592 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 02:40:45.500060 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.500039 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c7hfw\"" Jul 08 02:40:45.500210 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.500193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 02:40:45.500290 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.500199 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 02:40:45.505631 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.505607 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tdxfs"] Jul 08 02:40:45.510217 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.510182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8q9px"] Jul 08 02:40:45.682405 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.682375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6878310-383d-46ab-b566-005f4981c3f7-tmp-dir\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.682542 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.682462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.682542 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.682517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9q87\" (UniqueName: \"kubernetes.io/projected/d8462f8e-cbc6-4d23-8c1c-b99e3813824b-kube-api-access-l9q87\") pod \"ingress-canary-8q9px\" (UID: \"d8462f8e-cbc6-4d23-8c1c-b99e3813824b\") " pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.682627 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.682542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8462f8e-cbc6-4d23-8c1c-b99e3813824b-cert\") pod \"ingress-canary-8q9px\" (UID: \"d8462f8e-cbc6-4d23-8c1c-b99e3813824b\") " pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.682661 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.682630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccq7x\" (UniqueName: \"kubernetes.io/projected/a6878310-383d-46ab-b566-005f4981c3f7-kube-api-access-ccq7x\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.682694 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.682674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6878310-383d-46ab-b566-005f4981c3f7-config-volume\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.783656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.783616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.783791 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.783674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9q87\" (UniqueName: \"kubernetes.io/projected/d8462f8e-cbc6-4d23-8c1c-b99e3813824b-kube-api-access-l9q87\") pod \"ingress-canary-8q9px\" (UID: \"d8462f8e-cbc6-4d23-8c1c-b99e3813824b\") " pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.783791 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:45.783769 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:40:45.783859 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:45.783828 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:46.283808562 +0000 UTC m=+36.562185742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:40:45.783937 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.783918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8462f8e-cbc6-4d23-8c1c-b99e3813824b-cert\") pod \"ingress-canary-8q9px\" (UID: \"d8462f8e-cbc6-4d23-8c1c-b99e3813824b\") " pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.784001 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.783992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccq7x\" (UniqueName: \"kubernetes.io/projected/a6878310-383d-46ab-b566-005f4981c3f7-kube-api-access-ccq7x\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.784041 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.784026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6878310-383d-46ab-b566-005f4981c3f7-config-volume\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.784090 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.784055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6878310-383d-46ab-b566-005f4981c3f7-tmp-dir\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.784436 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.784415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6878310-383d-46ab-b566-005f4981c3f7-tmp-dir\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.784638 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.784618 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6878310-383d-46ab-b566-005f4981c3f7-config-volume\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.788142 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.788126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8462f8e-cbc6-4d23-8c1c-b99e3813824b-cert\") pod \"ingress-canary-8q9px\" (UID: \"d8462f8e-cbc6-4d23-8c1c-b99e3813824b\") " pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.792711 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.792693 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9q87\" (UniqueName: \"kubernetes.io/projected/d8462f8e-cbc6-4d23-8c1c-b99e3813824b-kube-api-access-l9q87\") pod \"ingress-canary-8q9px\" (UID: \"d8462f8e-cbc6-4d23-8c1c-b99e3813824b\") " pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.792791 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.792762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccq7x\" (UniqueName: \"kubernetes.io/projected/a6878310-383d-46ab-b566-005f4981c3f7-kube-api-access-ccq7x\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:45.811818 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.811799 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8q9px" Jul 08 02:40:45.956718 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:45.956495 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8q9px"] Jul 08 02:40:45.960592 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:40:45.960563 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8462f8e_cbc6_4d23_8c1c_b99e3813824b.slice/crio-b226ed569e07fe9596357851d10f7b6cad5d8a1df669d51a00bcbe0814ffad0b WatchSource:0}: Error finding container b226ed569e07fe9596357851d10f7b6cad5d8a1df669d51a00bcbe0814ffad0b: Status 404 returned error can't find the container with id b226ed569e07fe9596357851d10f7b6cad5d8a1df669d51a00bcbe0814ffad0b Jul 08 02:40:46.289445 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:46.289334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:46.290008 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:46.289486 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:40:46.290008 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:46.289545 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:47.289528933 +0000 UTC m=+37.567906132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:40:46.458067 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:46.458025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8q9px" event={"ID":"d8462f8e-cbc6-4d23-8c1c-b99e3813824b","Type":"ContainerStarted","Data":"b226ed569e07fe9596357851d10f7b6cad5d8a1df669d51a00bcbe0814ffad0b"} Jul 08 02:40:47.272555 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.272515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:40:47.272748 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.272518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:40:47.272748 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.272655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:40:47.275695 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.275670 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 02:40:47.276207 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.276179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 02:40:47.276633 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.276608 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 02:40:47.276633 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.276626 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 02:40:47.276633 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.276610 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2c2d\"" Jul 08 02:40:47.276851 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.276645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n5djs\"" Jul 08 02:40:47.298176 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:47.298153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:47.298544 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:47.298290 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:40:47.298544 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:47.298386 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:49.298367826 +0000 UTC m=+39.576745019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:40:48.466317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:48.466289 2576 generic.go:356] "Generic (PLEG): container finished" podID="7df7babb-e20c-4d79-8b15-6e7c34341630" containerID="715d9677e619efe777b95291d4222cc5e205be9bf688f1c8de02d8c0ae0dbb3c" exitCode=0 Jul 08 02:40:48.466824 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:48.466342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerDied","Data":"715d9677e619efe777b95291d4222cc5e205be9bf688f1c8de02d8c0ae0dbb3c"} Jul 08 02:40:49.311655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:49.311614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:49.311831 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:49.311781 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:40:49.311880 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:49.311845 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:40:53.311829875 +0000 UTC m=+43.590207055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:40:49.473314 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:49.473276 2576 generic.go:356] "Generic (PLEG): container finished" podID="7df7babb-e20c-4d79-8b15-6e7c34341630" containerID="d00249a34ee2da140e3e82d710f864ff6e7e152db82a02913c8de7997aafbee6" exitCode=0 Jul 08 02:40:49.473744 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:49.473317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerDied","Data":"d00249a34ee2da140e3e82d710f864ff6e7e152db82a02913c8de7997aafbee6"} Jul 08 02:40:50.479994 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.479742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fsp" event={"ID":"7df7babb-e20c-4d79-8b15-6e7c34341630","Type":"ContainerStarted","Data":"975742c45de24016d39502cee00ab0cb3f2bc17d71ec94b4b69294114fba915f"} Jul 08 02:40:50.481232 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.481206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8q9px" event={"ID":"d8462f8e-cbc6-4d23-8c1c-b99e3813824b","Type":"ContainerStarted","Data":"bc42dd51bc4f39b20ebf168bf0bb5394605c08de0529e63de5108a6cda2eeb85"} Jul 08 02:40:50.504183 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.504055 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-84fsp" podStartSLOduration=15.997806011 podStartE2EDuration="40.504040662s" totalImagesPullingTime="24.506234651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:50.502888407 +0000 UTC m=+40.781265622" watchObservedRunningTime="2026-07-08 02:40:50.504040662 +0000 UTC m=+40.782417864" Jul 08 02:40:50.522596 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.522554 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8q9px" podStartSLOduration=1.236504554 podStartE2EDuration="5.522538786s" totalImagesPullingTime="4.286034232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:40:50.521878231 +0000 UTC m=+40.800255433" watchObservedRunningTime="2026-07-08 02:40:50.522538786 +0000 UTC m=+40.800915989" Jul 08 02:40:50.623937 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.623900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:50.723702 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.723669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:50.865344 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.865277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:50.928987 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:50.928963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:51.034344 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:51.034315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:51.105072 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:51.105046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:52.809919 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:52.809882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:52.877729 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:52.877701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:40:53.340598 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:40:53.340565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:40:53.340769 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:53.340705 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:40:53.340769 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:40:53.340768 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:41:01.3407541 +0000 UTC m=+51.619131279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:41:01.034385 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:01.034336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:01.093588 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:01.093564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:01.396404 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:01.396375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:41:01.396533 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:01.396515 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:41:01.396592 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:01.396577 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:41:17.396560218 +0000 UTC m=+67.674937398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:41:10.449780 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:10.449752 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqdm8" Jul 08 02:41:10.599235 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:10.599204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:10.657480 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:10.657453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:12.949254 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:12.949225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:13.008189 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:13.008162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:13.069390 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:13.069343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:15.890797 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:15.890756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:41:15.893856 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:15.893834 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 02:41:15.901878 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:15.901861 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 02:41:15.901973 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:15.901932 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:42:19.901908679 +0000 UTC m=+130.180285861 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : secret "metrics-daemon-secret" not found Jul 08 02:41:15.991695 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:15.991659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:41:15.994240 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:15.994221 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 02:41:15.995724 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:15.995706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:41:16.005904 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.005884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e920314d-549a-4cd1-8160-4decdd56ad45-original-pull-secret\") pod \"global-pull-secret-syncer-w26sd\" (UID: \"e920314d-549a-4cd1-8160-4decdd56ad45\") " pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:41:16.084372 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.084322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w26sd" Jul 08 02:41:16.092597 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.092569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:41:16.095576 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.095551 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 02:41:16.105683 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.105659 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 02:41:16.116969 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.116937 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h68p\" (UniqueName: \"kubernetes.io/projected/a82ebec2-db66-4718-b77e-22d95f88692b-kube-api-access-4h68p\") pod \"network-check-target-kndhc\" (UID: \"a82ebec2-db66-4718-b77e-22d95f88692b\") " pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:41:16.226612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.226576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w26sd"] Jul 08 02:41:16.230211 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:41:16.230185 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode920314d_549a_4cd1_8160_4decdd56ad45.slice/crio-20f1fc913b66a22af880588c87764f3fd8ff927f67ee5fe0039cc294a0396105 WatchSource:0}: Error finding container 20f1fc913b66a22af880588c87764f3fd8ff927f67ee5fe0039cc294a0396105: Status 404 returned error can't find the container with id 20f1fc913b66a22af880588c87764f3fd8ff927f67ee5fe0039cc294a0396105 Jul 08 02:41:16.400729 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.400697 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2c2d\"" Jul 08 02:41:16.408655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.408628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:41:16.527728 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.527698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kndhc"] Jul 08 02:41:16.530876 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:41:16.530835 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82ebec2_db66_4718_b77e_22d95f88692b.slice/crio-840ca50c63d2ed0c3c67f2e3b57279bd52dfddb115bf17177a759e94ee226e8a WatchSource:0}: Error finding container 840ca50c63d2ed0c3c67f2e3b57279bd52dfddb115bf17177a759e94ee226e8a: Status 404 returned error can't find the container with id 840ca50c63d2ed0c3c67f2e3b57279bd52dfddb115bf17177a759e94ee226e8a Jul 08 02:41:16.541092 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.541068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w26sd" event={"ID":"e920314d-549a-4cd1-8160-4decdd56ad45","Type":"ContainerStarted","Data":"20f1fc913b66a22af880588c87764f3fd8ff927f67ee5fe0039cc294a0396105"} Jul 08 02:41:16.542072 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:16.542052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kndhc" event={"ID":"a82ebec2-db66-4718-b77e-22d95f88692b","Type":"ContainerStarted","Data":"840ca50c63d2ed0c3c67f2e3b57279bd52dfddb115bf17177a759e94ee226e8a"} Jul 08 02:41:17.402086 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:17.402047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:41:17.402557 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:17.402227 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:41:17.402557 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:17.402287 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:41:49.402270222 +0000 UTC m=+99.680647419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:41:21.558876 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:21.558779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w26sd" event={"ID":"e920314d-549a-4cd1-8160-4decdd56ad45","Type":"ContainerStarted","Data":"63d3da89d1158ab556a701ea90abf8d84eacb5b1cb15a9ccf2b3faa9b79f7eda"} Jul 08 02:41:21.560311 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:21.560289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kndhc" event={"ID":"a82ebec2-db66-4718-b77e-22d95f88692b","Type":"ContainerStarted","Data":"e16d56a71fb24c816dbfeca79a00b9c65ccceb6affe817b53c62511faad9951d"} Jul 08 02:41:21.560430 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:21.560414 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:41:21.574054 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:21.574013 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w26sd" podStartSLOduration=65.642770334 podStartE2EDuration="1m10.574000058s" totalImagesPullingTime="4.931229724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:41:21.572933078 +0000 UTC m=+71.851310279" watchObservedRunningTime="2026-07-08 02:41:21.574000058 +0000 UTC m=+71.852377260" Jul 08 02:41:21.586484 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:21.586446 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kndhc" podStartSLOduration=66.957244741 podStartE2EDuration="1m11.586433217s" totalImagesPullingTime="4.629188476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:41:21.586243668 +0000 UTC m=+71.864620882" watchObservedRunningTime="2026-07-08 02:41:21.586433217 +0000 UTC m=+71.864810416" Jul 08 02:41:30.689517 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.689403 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825"] Jul 08 02:41:30.691339 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.691322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:30.694593 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.694567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m9rv7\"" Jul 08 02:41:30.694593 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.694576 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 02:41:30.694593 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.694567 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 02:41:30.694806 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.694582 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 02:41:30.694806 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.694582 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 02:41:30.701768 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.701743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825"] Jul 08 02:41:30.791943 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.791913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd0acbc6-9373-4468-a09b-b968fa67ab36-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5899899db-5b825\" (UID: \"bd0acbc6-9373-4468-a09b-b968fa67ab36\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:30.792063 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.791961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfwx\" (UniqueName: \"kubernetes.io/projected/bd0acbc6-9373-4468-a09b-b968fa67ab36-kube-api-access-cdfwx\") pod \"managed-serviceaccount-addon-agent-5899899db-5b825\" (UID: \"bd0acbc6-9373-4468-a09b-b968fa67ab36\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:30.892426 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.892396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd0acbc6-9373-4468-a09b-b968fa67ab36-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5899899db-5b825\" (UID: \"bd0acbc6-9373-4468-a09b-b968fa67ab36\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:30.892513 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.892449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfwx\" (UniqueName: \"kubernetes.io/projected/bd0acbc6-9373-4468-a09b-b968fa67ab36-kube-api-access-cdfwx\") pod \"managed-serviceaccount-addon-agent-5899899db-5b825\" (UID: \"bd0acbc6-9373-4468-a09b-b968fa67ab36\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:30.894958 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.894934 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd0acbc6-9373-4468-a09b-b968fa67ab36-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5899899db-5b825\" (UID: \"bd0acbc6-9373-4468-a09b-b968fa67ab36\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:30.900982 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:30.900959 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfwx\" (UniqueName: \"kubernetes.io/projected/bd0acbc6-9373-4468-a09b-b968fa67ab36-kube-api-access-cdfwx\") pod \"managed-serviceaccount-addon-agent-5899899db-5b825\" (UID: \"bd0acbc6-9373-4468-a09b-b968fa67ab36\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:31.011166 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:31.011098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:33.595787 2576 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74" Netns:"/var/run/netns/af9188c5-2a3b-42ab-beed-6d93ae000af1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36" Path:"" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get "https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s": context deadline exceeded Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: > Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:33.595860 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74" Netns:"/var/run/netns/af9188c5-2a3b-42ab-beed-6d93ae000af1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36" Path:"" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get "https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s": context deadline exceeded Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: > pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:33.595878 2576 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 08 02:41:33.596010 ip-10-0-128-107 kubenswrapper[2576]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74" Netns:"/var/run/netns/af9188c5-2a3b-42ab-beed-6d93ae000af1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36" Path:"" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get "https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s": context deadline exceeded Jul 08 02:41:33.596646 ip-10-0-128-107 kubenswrapper[2576]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 08 02:41:33.596646 ip-10-0-128-107 kubenswrapper[2576]: > pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:33.596646 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:33.595942 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon(bd0acbc6-9373-4468-a09b-b968fa67ab36)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon(bd0acbc6-9373-4468-a09b-b968fa67ab36)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74\\\" Netns:\\\"/var/run/netns/af9188c5-2a3b-42ab-beed-6d93ae000af1\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=f496e4d77a06012095ef7651aeafa3bcf62e11c48cd7e57e0fab48b6f43c7f74;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36\\\" Path:\\\"\\\" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get \\\"https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" podUID="bd0acbc6-9373-4468-a09b-b968fa67ab36" Jul 08 02:41:34.588133 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:34.588100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:34.593391 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:34.593373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:37.178898 2576 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3" Netns:"/var/run/netns/22ad1965-36c4-4b64-9564-8251ebe51995" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36" Path:"" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get "https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s": context deadline exceeded Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: > Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:37.178944 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3" Netns:"/var/run/netns/22ad1965-36c4-4b64-9564-8251ebe51995" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36" Path:"" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get "https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s": context deadline exceeded Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: > pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:37.178961 2576 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 08 02:41:37.179119 ip-10-0-128-107 kubenswrapper[2576]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3" Netns:"/var/run/netns/22ad1965-36c4-4b64-9564-8251ebe51995" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36" Path:"" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get "https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s": context deadline exceeded Jul 08 02:41:37.179753 ip-10-0-128-107 kubenswrapper[2576]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 08 02:41:37.179753 ip-10-0-128-107 kubenswrapper[2576]: > pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:37.179753 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:37.179060 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon(bd0acbc6-9373-4468-a09b-b968fa67ab36)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon(bd0acbc6-9373-4468-a09b-b968fa67ab36)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_managed-serviceaccount-addon-agent-5899899db-5b825_open-cluster-management-agent-addon_bd0acbc6-9373-4468-a09b-b968fa67ab36_0(63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3): error adding pod open-cluster-management-agent-addon_managed-serviceaccount-addon-agent-5899899db-5b825 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3\\\" Netns:\\\"/var/run/netns/22ad1965-36c4-4b64-9564-8251ebe51995\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=open-cluster-management-agent-addon;K8S_POD_NAME=managed-serviceaccount-addon-agent-5899899db-5b825;K8S_POD_INFRA_CONTAINER_ID=63875b1d15d4a27ca9f60c70fc812ddfef4875bd10a3e34bad2a8f41089b55a3;K8S_POD_UID=bd0acbc6-9373-4468-a09b-b968fa67ab36\\\" Path:\\\"\\\" ERRORED: error configuring pod [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825] networking: Multus: [open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825/bd0acbc6-9373-4468-a09b-b968fa67ab36]: error waiting for pod: Get \\\"https://a4536e496c57f484fb86c43d48887d39-dd8d149628a69fcd.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/open-cluster-management-agent-addon/pods/managed-serviceaccount-addon-agent-5899899db-5b825?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" podUID="bd0acbc6-9373-4468-a09b-b968fa67ab36" Jul 08 02:41:47.272454 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:47.272408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:47.272869 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:47.272769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" Jul 08 02:41:49.256785 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:49.256742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825"] Jul 08 02:41:49.262594 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:41:49.262567 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd0acbc6_9373_4468_a09b_b968fa67ab36.slice/crio-8d11ea8f9498d1a6f93db7b4168eeb909e53bad38b4da572d540881f1edacb61 WatchSource:0}: Error finding container 8d11ea8f9498d1a6f93db7b4168eeb909e53bad38b4da572d540881f1edacb61: Status 404 returned error can't find the container with id 8d11ea8f9498d1a6f93db7b4168eeb909e53bad38b4da572d540881f1edacb61 Jul 08 02:41:49.404821 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:49.404794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:41:49.404974 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:49.404933 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:41:49.405013 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:41:49.404990 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:42:53.404975073 +0000 UTC m=+163.683352254 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:41:49.627819 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:49.627780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" event={"ID":"bd0acbc6-9373-4468-a09b-b968fa67ab36","Type":"ContainerStarted","Data":"8d11ea8f9498d1a6f93db7b4168eeb909e53bad38b4da572d540881f1edacb61"} Jul 08 02:41:52.564593 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:52.564557 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kndhc" Jul 08 02:41:53.639494 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:53.639461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" event={"ID":"bd0acbc6-9373-4468-a09b-b968fa67ab36","Type":"ContainerStarted","Data":"f7ae3bd5d07b0b69079ff9ff2067d51bca49d4243e57d3a0bb714100cba725c3"} Jul 08 02:41:53.657144 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:41:53.656986 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" podStartSLOduration=20.208893116 podStartE2EDuration="23.656968367s" totalImagesPullingTime="3.448075251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:41:53.656784463 +0000 UTC m=+103.935161678" watchObservedRunningTime="2026-07-08 02:41:53.656968367 +0000 UTC m=+103.935345571" Jul 08 02:42:19.905021 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:42:19.904984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:42:19.905575 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:42:19.905140 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 02:42:19.905575 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:42:19.905262 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs podName:4e669e30-a095-4fe7-be16-7ca17010a100 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:21.905241081 +0000 UTC m=+252.183618283 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs") pod "network-metrics-daemon-frgkz" (UID: "4e669e30-a095-4fe7-be16-7ca17010a100") : secret "metrics-daemon-secret" not found Jul 08 02:42:37.575571 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:42:37.575536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:42:48.503769 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:42:48.503728 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tdxfs" podUID="a6878310-383d-46ab-b566-005f4981c3f7" Jul 08 02:42:48.764476 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:42:48.764409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdxfs" Jul 08 02:42:50.291292 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:42:50.291258 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-frgkz" podUID="4e669e30-a095-4fe7-be16-7ca17010a100" Jul 08 02:42:53.416871 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:42:53.416824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:42:53.417267 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:42:53.416992 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:42:53.417267 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:42:53.417058 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls podName:a6878310-383d-46ab-b566-005f4981c3f7 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:55.417040913 +0000 UTC m=+285.695418093 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls") pod "dns-default-tdxfs" (UID: "a6878310-383d-46ab-b566-005f4981c3f7") : secret "dns-default-metrics-tls" not found Jul 08 02:43:03.272158 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:43:03.272078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:43:36.249915 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:43:36.249881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:43:52.911189 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:43:52.911160 2576 generic.go:356] "Generic (PLEG): container finished" podID="bd0acbc6-9373-4468-a09b-b968fa67ab36" containerID="f7ae3bd5d07b0b69079ff9ff2067d51bca49d4243e57d3a0bb714100cba725c3" exitCode=255 Jul 08 02:43:52.911516 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:43:52.911194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" event={"ID":"bd0acbc6-9373-4468-a09b-b968fa67ab36","Type":"ContainerDied","Data":"f7ae3bd5d07b0b69079ff9ff2067d51bca49d4243e57d3a0bb714100cba725c3"} Jul 08 02:43:52.911516 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:43:52.911474 2576 scope.go:117] "RemoveContainer" containerID="f7ae3bd5d07b0b69079ff9ff2067d51bca49d4243e57d3a0bb714100cba725c3" Jul 08 02:43:53.915629 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:43:53.915599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5899899db-5b825" event={"ID":"bd0acbc6-9373-4468-a09b-b968fa67ab36","Type":"ContainerStarted","Data":"1814b4ca5b90a28ec62f494207f5f87410fd0d9827859da2957826f9ac031c0a"} Jul 08 02:44:02.360838 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.360806 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f8b685459-ftl64"] Jul 08 02:44:02.363795 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.363778 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk"] Jul 08 02:44:02.363919 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.363904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.366894 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.366871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:02.372758 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.372736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 02:44:02.372863 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.372761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 02:44:02.372863 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.372802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 02:44:02.372976 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.372858 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 02:44:02.372976 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.372865 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 02:44:02.373085 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.373068 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 02:44:02.373144 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.373091 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tqcmf\"" Jul 08 02:44:02.373189 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.373165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 02:44:02.373257 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.373242 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2lb26\"" Jul 08 02:44:02.373516 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.373501 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:44:02.373788 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.373772 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 02:44:02.388836 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.388814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk"] Jul 08 02:44:02.389623 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.389604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f8b685459-ftl64"] Jul 08 02:44:02.453635 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-stats-auth\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.453747 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.453747 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:02.453747 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.453747 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxb4\" (UniqueName: \"kubernetes.io/projected/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-kube-api-access-jkxb4\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.453877 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbnf\" (UniqueName: \"kubernetes.io/projected/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-kube-api-access-bnbnf\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:02.453877 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.453787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-default-certificate\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.460086 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.460064 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-786pp"] Jul 08 02:44:02.462788 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.462774 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wphwj"] Jul 08 02:44:02.462909 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.462891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.465192 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.465174 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 02:44:02.465268 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.465229 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d66d495bb-fgs92"] Jul 08 02:44:02.465393 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.465377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.465667 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.465645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m529r\"" Jul 08 02:44:02.466011 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.465995 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 02:44:02.466281 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.466267 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 02:44:02.467758 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.467734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.468632 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.468604 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 02:44:02.468758 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.468736 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 02:44:02.468912 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.468897 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 02:44:02.469477 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.469459 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 02:44:02.469477 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.469474 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:44:02.469594 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.469485 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d84rs\"" Jul 08 02:44:02.469926 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.469901 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 02:44:02.470564 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.470547 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 02:44:02.470812 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.470670 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 02:44:02.471458 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.471433 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tthw\"" Jul 08 02:44:02.483134 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.481966 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 02:44:02.483490 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.483375 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-786pp"] Jul 08 02:44:02.484564 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.484544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wphwj"] Jul 08 02:44:02.484665 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.484580 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 02:44:02.485922 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.485902 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 02:44:02.486007 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.485991 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d66d495bb-fgs92"] Jul 08 02:44:02.554726 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-stats-auth\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.554726 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b1e0477-6102-4d78-8329-f3792ea19ef5-tmp\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1e0477-6102-4d78-8329-f3792ea19ef5-serving-cert\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlss7\" (UniqueName: \"kubernetes.io/projected/7b1e0477-6102-4d78-8329-f3792ea19ef5-kube-api-access-zlss7\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-image-registry-private-configuration\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b1e0477-6102-4d78-8329-f3792ea19ef5-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:02.554946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.554940 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.554966 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:03.054950925 +0000 UTC m=+233.333328106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : configmap references non-existent config key: service-ca.crt Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxb4\" (UniqueName: \"kubernetes.io/projected/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-kube-api-access-jkxb4\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.554998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-ca-trust-extracted\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.555016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls podName:5becb78c-aca9-4cd2-939b-3a5d94c1f9d2 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:03.054997954 +0000 UTC m=+233.333375138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jczdk" (UID: "5becb78c-aca9-4cd2-939b-3a5d94c1f9d2") : secret "samples-operator-tls" not found Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbnf\" (UniqueName: \"kubernetes.io/projected/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-kube-api-access-bnbnf\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.555073 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-default-certificate\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.555106 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:03.055095156 +0000 UTC m=+233.333472339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : secret "router-metrics-certs-default" not found Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b1e0477-6102-4d78-8329-f3792ea19ef5-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/959b648d-524d-4fb4-961b-b32f222717ee-config\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-installation-pull-secrets\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b1e0477-6102-4d78-8329-f3792ea19ef5-snapshots\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/959b648d-524d-4fb4-961b-b32f222717ee-serving-cert\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.555459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.555955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4gz\" (UniqueName: \"kubernetes.io/projected/959b648d-524d-4fb4-961b-b32f222717ee-kube-api-access-xj4gz\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.555955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/959b648d-524d-4fb4-961b-b32f222717ee-trusted-ca\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.555955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-bound-sa-token\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.555955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-trusted-ca\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.555955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-certificates\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.555955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.555678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8mz\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-kube-api-access-5t8mz\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.557238 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.557218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-stats-auth\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.557449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.557431 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-default-certificate\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.565936 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.565912 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbnf\" (UniqueName: \"kubernetes.io/projected/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-kube-api-access-bnbnf\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:02.566023 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.565917 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxb4\" (UniqueName: \"kubernetes.io/projected/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-kube-api-access-jkxb4\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:02.604591 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.604564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:02.656856 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-ca-trust-extracted\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.656942 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b1e0477-6102-4d78-8329-f3792ea19ef5-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.656942 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/959b648d-524d-4fb4-961b-b32f222717ee-config\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.656942 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-installation-pull-secrets\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.656942 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b1e0477-6102-4d78-8329-f3792ea19ef5-snapshots\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.657143 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/959b648d-524d-4fb4-961b-b32f222717ee-serving-cert\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.657143 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.656968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657143 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4gz\" (UniqueName: \"kubernetes.io/projected/959b648d-524d-4fb4-961b-b32f222717ee-kube-api-access-xj4gz\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.657143 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/959b648d-524d-4fb4-961b-b32f222717ee-trusted-ca\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.657143 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-bound-sa-token\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-trusted-ca\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-certificates\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8mz\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-kube-api-access-5t8mz\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657270 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-ca-trust-extracted\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b1e0477-6102-4d78-8329-f3792ea19ef5-tmp\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1e0477-6102-4d78-8329-f3792ea19ef5-serving-cert\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlss7\" (UniqueName: \"kubernetes.io/projected/7b1e0477-6102-4d78-8329-f3792ea19ef5-kube-api-access-zlss7\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-image-registry-private-configuration\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.657473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b1e0477-6102-4d78-8329-f3792ea19ef5-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.658073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b1e0477-6102-4d78-8329-f3792ea19ef5-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.658073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.657669 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/959b648d-524d-4fb4-961b-b32f222717ee-config\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.658181 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.658079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-certificates\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.658472 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.658442 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b1e0477-6102-4d78-8329-f3792ea19ef5-tmp\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.658758 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.658684 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:44:02.658758 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.658700 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66d495bb-fgs92: secret "image-registry-tls" not found Jul 08 02:44:02.659036 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:02.658761 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls podName:4f8f3b8d-2980-45e0-8994-4b7f0e3fa629 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:03.158745528 +0000 UTC m=+233.437122708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls") pod "image-registry-7d66d495bb-fgs92" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629") : secret "image-registry-tls" not found Jul 08 02:44:02.659036 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.658964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b1e0477-6102-4d78-8329-f3792ea19ef5-snapshots\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.659201 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.659114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b1e0477-6102-4d78-8329-f3792ea19ef5-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.659319 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.659296 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-trusted-ca\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.659724 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.659703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/959b648d-524d-4fb4-961b-b32f222717ee-trusted-ca\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.659924 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.659907 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-installation-pull-secrets\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.659990 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.659909 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/959b648d-524d-4fb4-961b-b32f222717ee-serving-cert\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.660576 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.660549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-image-registry-private-configuration\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.661123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.661105 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1e0477-6102-4d78-8329-f3792ea19ef5-serving-cert\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.666867 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.666839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-bound-sa-token\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.666965 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.666950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8mz\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-kube-api-access-5t8mz\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:02.667768 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.667750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlss7\" (UniqueName: \"kubernetes.io/projected/7b1e0477-6102-4d78-8329-f3792ea19ef5-kube-api-access-zlss7\") pod \"insights-operator-5c847dcfc4-wphwj\" (UID: \"7b1e0477-6102-4d78-8329-f3792ea19ef5\") " pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.667859 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.667843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4gz\" (UniqueName: \"kubernetes.io/projected/959b648d-524d-4fb4-961b-b32f222717ee-kube-api-access-xj4gz\") pod \"console-operator-68b757fb67-786pp\" (UID: \"959b648d-524d-4fb4-961b-b32f222717ee\") " pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.705995 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.705968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:02.774262 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.774232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:02.787491 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.787474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" Jul 08 02:44:02.933579 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.933509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-786pp"] Jul 08 02:44:02.943124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:02.943105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wphwj"] Jul 08 02:44:02.945481 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:02.945457 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b648d_524d_4fb4_961b_b32f222717ee.slice/crio-3292d0773a2f6b13b891412c9bafcb3331bddd1ee871d0dbd0a55d0457885b6e WatchSource:0}: Error finding container 3292d0773a2f6b13b891412c9bafcb3331bddd1ee871d0dbd0a55d0457885b6e: Status 404 returned error can't find the container with id 3292d0773a2f6b13b891412c9bafcb3331bddd1ee871d0dbd0a55d0457885b6e Jul 08 02:44:02.952093 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:02.952072 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1e0477_6102_4d78_8329_f3792ea19ef5.slice/crio-72f7e7d86b8454934971d3f0941ec677ec017bb94efed7df36aa7fd264d8fcd1 WatchSource:0}: Error finding container 72f7e7d86b8454934971d3f0941ec677ec017bb94efed7df36aa7fd264d8fcd1: Status 404 returned error can't find the container with id 72f7e7d86b8454934971d3f0941ec677ec017bb94efed7df36aa7fd264d8fcd1 Jul 08 02:44:03.062273 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:03.062247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:03.062374 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:03.062278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:03.062374 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:03.062299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:03.062456 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.062419 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:44:03.062456 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.062442 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:04.062426051 +0000 UTC m=+234.340803235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : configmap references non-existent config key: service-ca.crt Jul 08 02:44:03.062523 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.062464 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls podName:5becb78c-aca9-4cd2-939b-3a5d94c1f9d2 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:04.06245171 +0000 UTC m=+234.340828890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jczdk" (UID: "5becb78c-aca9-4cd2-939b-3a5d94c1f9d2") : secret "samples-operator-tls" not found Jul 08 02:44:03.062523 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.062462 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:44:03.062588 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.062540 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:04.062523013 +0000 UTC m=+234.340900211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : secret "router-metrics-certs-default" not found Jul 08 02:44:03.163685 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:03.163660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:03.163779 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.163749 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:44:03.163779 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.163758 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66d495bb-fgs92: secret "image-registry-tls" not found Jul 08 02:44:03.163855 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:03.163796 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls podName:4f8f3b8d-2980-45e0-8994-4b7f0e3fa629 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:04.163784614 +0000 UTC m=+234.442161794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls") pod "image-registry-7d66d495bb-fgs92" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629") : secret "image-registry-tls" not found Jul 08 02:44:03.941583 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:03.941543 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" event={"ID":"7b1e0477-6102-4d78-8329-f3792ea19ef5","Type":"ContainerStarted","Data":"72f7e7d86b8454934971d3f0941ec677ec017bb94efed7df36aa7fd264d8fcd1"} Jul 08 02:44:03.942907 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:03.942876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-786pp" event={"ID":"959b648d-524d-4fb4-961b-b32f222717ee","Type":"ContainerStarted","Data":"3292d0773a2f6b13b891412c9bafcb3331bddd1ee871d0dbd0a55d0457885b6e"} Jul 08 02:44:04.070160 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:04.070134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:04.070336 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:04.070166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:04.070336 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:04.070188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:04.070336 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.070296 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:44:04.070336 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.070298 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:44:04.070530 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.070309 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:06.070290145 +0000 UTC m=+236.348667332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : configmap references non-existent config key: service-ca.crt Jul 08 02:44:04.070530 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.070386 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:06.070349608 +0000 UTC m=+236.348726803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : secret "router-metrics-certs-default" not found Jul 08 02:44:04.070530 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.070402 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls podName:5becb78c-aca9-4cd2-939b-3a5d94c1f9d2 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:06.070393098 +0000 UTC m=+236.348770278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jczdk" (UID: "5becb78c-aca9-4cd2-939b-3a5d94c1f9d2") : secret "samples-operator-tls" not found Jul 08 02:44:04.171632 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:04.171593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:04.171798 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.171710 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:44:04.171798 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.171725 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66d495bb-fgs92: secret "image-registry-tls" not found Jul 08 02:44:04.171798 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:04.171782 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls podName:4f8f3b8d-2980-45e0-8994-4b7f0e3fa629 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:06.171768777 +0000 UTC m=+236.450145957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls") pod "image-registry-7d66d495bb-fgs92" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629") : secret "image-registry-tls" not found Jul 08 02:44:04.486036 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:04.485985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:04.542679 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:04.542641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:05.954696 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:05.954664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" event={"ID":"7b1e0477-6102-4d78-8329-f3792ea19ef5","Type":"ContainerStarted","Data":"256de28a5e88666ba65e6faa410228b226566bae9b2bece03d7fa1681feb4289"} Jul 08 02:44:05.956302 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:05.956284 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/0.log" Jul 08 02:44:05.956412 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:05.956318 2576 generic.go:356] "Generic (PLEG): container finished" podID="959b648d-524d-4fb4-961b-b32f222717ee" containerID="18a95be2bceba02a197c2c4b357db81b17ea6602187aface9348168809ecdaf3" exitCode=255 Jul 08 02:44:05.956412 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:05.956339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-786pp" event={"ID":"959b648d-524d-4fb4-961b-b32f222717ee","Type":"ContainerDied","Data":"18a95be2bceba02a197c2c4b357db81b17ea6602187aface9348168809ecdaf3"} Jul 08 02:44:05.956571 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:05.956557 2576 scope.go:117] "RemoveContainer" containerID="18a95be2bceba02a197c2c4b357db81b17ea6602187aface9348168809ecdaf3" Jul 08 02:44:05.970250 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:05.970196 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" podStartSLOduration=1.354450166 podStartE2EDuration="3.970180288s" totalImagesPullingTime="2.615730122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:05.969427547 +0000 UTC m=+236.247804751" watchObservedRunningTime="2026-07-08 02:44:05.970180288 +0000 UTC m=+236.248557492" Jul 08 02:44:06.086319 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.086290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:06.086463 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.086325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:06.086463 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.086374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:06.086575 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.086480 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:10.086459568 +0000 UTC m=+240.364836768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : configmap references non-existent config key: service-ca.crt Jul 08 02:44:06.086999 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.086789 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:44:06.086999 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.086843 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls podName:5becb78c-aca9-4cd2-939b-3a5d94c1f9d2 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:10.086827847 +0000 UTC m=+240.365205033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jczdk" (UID: "5becb78c-aca9-4cd2-939b-3a5d94c1f9d2") : secret "samples-operator-tls" not found Jul 08 02:44:06.086999 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.086909 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:44:06.086999 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.086947 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:10.086933805 +0000 UTC m=+240.365310989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : secret "router-metrics-certs-default" not found Jul 08 02:44:06.165146 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.165121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:06.186952 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.186930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:06.187073 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.187059 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:44:06.187130 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.187074 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66d495bb-fgs92: secret "image-registry-tls" not found Jul 08 02:44:06.187130 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.187121 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls podName:4f8f3b8d-2980-45e0-8994-4b7f0e3fa629 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:10.187103633 +0000 UTC m=+240.465480814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls") pod "image-registry-7d66d495bb-fgs92" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629") : secret "image-registry-tls" not found Jul 08 02:44:06.594318 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.594286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:06.646073 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.646040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:06.898936 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.898912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:06.962696 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.962594 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:44:06.964208 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.963863 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/0.log" Jul 08 02:44:06.964208 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.963911 2576 generic.go:356] "Generic (PLEG): container finished" podID="959b648d-524d-4fb4-961b-b32f222717ee" containerID="0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda" exitCode=255 Jul 08 02:44:06.964208 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.964008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-786pp" event={"ID":"959b648d-524d-4fb4-961b-b32f222717ee","Type":"ContainerDied","Data":"0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda"} Jul 08 02:44:06.964208 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.964145 2576 scope.go:117] "RemoveContainer" containerID="18a95be2bceba02a197c2c4b357db81b17ea6602187aface9348168809ecdaf3" Jul 08 02:44:06.964802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.964532 2576 scope.go:117] "RemoveContainer" containerID="0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda" Jul 08 02:44:06.964802 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:06.964737 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-786pp_openshift-console-operator(959b648d-524d-4fb4-961b-b32f222717ee)\"" pod="openshift-console-operator/console-operator-68b757fb67-786pp" podUID="959b648d-524d-4fb4-961b-b32f222717ee" Jul 08 02:44:06.982422 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.982346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:06.988864 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.988844 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hjnfq"] Jul 08 02:44:06.993000 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.992983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:06.995616 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.995594 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 02:44:06.995722 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.995615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 02:44:06.995722 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:06.995638 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6xd4b\"" Jul 08 02:44:07.003384 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.003346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjnfq"] Jul 08 02:44:07.044634 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.044604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:07.094140 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.094113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-data-volume\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.094262 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.094209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.094262 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.094243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp74q\" (UniqueName: \"kubernetes.io/projected/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-api-access-xp74q\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.094400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.094273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-crio-socket\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.094466 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.094450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.195780 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.195727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.195780 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.195761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp74q\" (UniqueName: \"kubernetes.io/projected/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-api-access-xp74q\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.195900 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.195782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-crio-socket\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.195900 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.195833 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:44:07.195900 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.195884 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:07.695867847 +0000 UTC m=+237.974245027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : configmap "kube-rbac-proxy" not found Jul 08 02:44:07.196026 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.195918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-crio-socket\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.196026 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.195941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.196026 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.195972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-data-volume\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.196162 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.196093 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:44:07.196162 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.196153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:07.696136527 +0000 UTC m=+237.974513714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : secret "insights-runtime-extractor-tls" not found Jul 08 02:44:07.196279 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.196212 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-data-volume\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.203891 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.203871 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp74q\" (UniqueName: \"kubernetes.io/projected/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-api-access-xp74q\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.304182 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.304154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:07.403439 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.403411 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:07.503540 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.503487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:07.699683 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.699651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.699847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.699716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:07.699847 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.699789 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:44:07.699916 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.699853 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:08.699839068 +0000 UTC m=+238.978216248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : configmap "kube-rbac-proxy" not found Jul 08 02:44:07.699916 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.699797 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:44:07.699916 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.699910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:08.699898987 +0000 UTC m=+238.978276167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : secret "insights-runtime-extractor-tls" not found Jul 08 02:44:07.968674 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.968649 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:44:07.969038 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:07.968970 2576 scope.go:117] "RemoveContainer" containerID="0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda" Jul 08 02:44:07.969130 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:07.969111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-786pp_openshift-console-operator(959b648d-524d-4fb4-961b-b32f222717ee)\"" pod="openshift-console-operator/console-operator-68b757fb67-786pp" podUID="959b648d-524d-4fb4-961b-b32f222717ee" Jul 08 02:44:08.465429 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.465402 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-9j9r9"] Jul 08 02:44:08.468656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.468636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.471446 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.471427 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 02:44:08.471562 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.471491 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 02:44:08.472204 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.472187 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-l654f\"" Jul 08 02:44:08.472297 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.472187 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 02:44:08.472297 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.472237 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 02:44:08.477736 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.477715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-9j9r9"] Jul 08 02:44:08.505386 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.505342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcvz\" (UniqueName: \"kubernetes.io/projected/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-kube-api-access-lzcvz\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.505497 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.505400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-signing-key\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.505545 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.505504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-signing-cabundle\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.507016 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.506990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:08.560304 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.560276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:08.606802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.606774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcvz\" (UniqueName: \"kubernetes.io/projected/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-kube-api-access-lzcvz\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.606933 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.606819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-signing-key\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.607078 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.607058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-signing-cabundle\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.607617 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.607599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-signing-cabundle\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.609400 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.609379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-signing-key\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.614939 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.614910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcvz\" (UniqueName: \"kubernetes.io/projected/d40ed506-82b2-4ec5-b0a9-8db3dbc8265f-kube-api-access-lzcvz\") pod \"service-ca-758f599c5-9j9r9\" (UID: \"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f\") " pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.707949 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.707901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:08.708122 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.708009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:08.708122 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:08.708031 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:44:08.708122 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:08.708090 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:10.708076104 +0000 UTC m=+240.986453284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : configmap "kube-rbac-proxy" not found Jul 08 02:44:08.708280 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:08.708137 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:44:08.708280 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:08.708194 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:10.708178505 +0000 UTC m=+240.986555688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : secret "insights-runtime-extractor-tls" not found Jul 08 02:44:08.775770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.775699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5s2w_ba220b2f-2b74-4163-a98f-9975b0a172f5/dns-node-resolver/0.log" Jul 08 02:44:08.777844 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.777823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-9j9r9" Jul 08 02:44:08.898604 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.898574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-9j9r9"] Jul 08 02:44:08.903692 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:08.903667 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40ed506_82b2_4ec5_b0a9_8db3dbc8265f.slice/crio-a77ca589b86e3a05394dd901fa2ad823221ada60a9c27721258b63a5ee74b4dc WatchSource:0}: Error finding container a77ca589b86e3a05394dd901fa2ad823221ada60a9c27721258b63a5ee74b4dc: Status 404 returned error can't find the container with id a77ca589b86e3a05394dd901fa2ad823221ada60a9c27721258b63a5ee74b4dc Jul 08 02:44:08.972921 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:08.972883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-9j9r9" event={"ID":"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f","Type":"ContainerStarted","Data":"a77ca589b86e3a05394dd901fa2ad823221ada60a9c27721258b63a5ee74b4dc"} Jul 08 02:44:09.368944 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:09.368900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:10.120018 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.119991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:10.120018 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.120021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:10.120496 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.120052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:10.120496 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.120141 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:18.120125352 +0000 UTC m=+248.398502532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : configmap references non-existent config key: service-ca.crt Jul 08 02:44:10.120496 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.120200 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:44:10.120496 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.120218 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:44:10.120496 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.120256 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:18.120240438 +0000 UTC m=+248.398617620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : secret "router-metrics-certs-default" not found Jul 08 02:44:10.120496 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.120274 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls podName:5becb78c-aca9-4cd2-939b-3a5d94c1f9d2 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:18.120264281 +0000 UTC m=+248.398641462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jczdk" (UID: "5becb78c-aca9-4cd2-939b-3a5d94c1f9d2") : secret "samples-operator-tls" not found Jul 08 02:44:10.176211 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.176183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vfjph_010a311a-d396-497b-9742-78f12c75dfd8/node-ca/0.log" Jul 08 02:44:10.221461 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.221426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:10.221617 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.221566 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:44:10.221617 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.221582 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66d495bb-fgs92: secret "image-registry-tls" not found Jul 08 02:44:10.221716 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.221645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls podName:4f8f3b8d-2980-45e0-8994-4b7f0e3fa629 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:18.221624487 +0000 UTC m=+248.500001757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls") pod "image-registry-7d66d495bb-fgs92" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629") : secret "image-registry-tls" not found Jul 08 02:44:10.577156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.577125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8q9px_d8462f8e-cbc6-4d23-8c1c-b99e3813824b/serve-healthcheck-canary/0.log" Jul 08 02:44:10.725118 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.725079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:10.725310 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.725198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:10.725310 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.725258 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:44:10.725441 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.725338 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:14.725316149 +0000 UTC m=+245.003693333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : configmap "kube-rbac-proxy" not found Jul 08 02:44:10.725441 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.725345 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:44:10.725441 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:10.725405 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:14.725392177 +0000 UTC m=+245.003769357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : secret "insights-runtime-extractor-tls" not found Jul 08 02:44:10.981140 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.981110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-9j9r9" event={"ID":"d40ed506-82b2-4ec5-b0a9-8db3dbc8265f","Type":"ContainerStarted","Data":"6564fe1de7982bf57de80874978d9d6e7ebe7cdc05171b8d3ee233483f6c30c9"} Jul 08 02:44:10.997130 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:10.997085 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-9j9r9" podStartSLOduration=1.059818291 podStartE2EDuration="2.997070524s" totalImagesPullingTime="1.937252233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:10.996200227 +0000 UTC m=+241.274577445" watchObservedRunningTime="2026-07-08 02:44:10.997070524 +0000 UTC m=+241.275447725" Jul 08 02:44:12.278019 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.277987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:12.369864 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.369820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:12.554898 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.554788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:12.622053 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.622028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:12.749689 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.749660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:12.774457 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.774436 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:12.774588 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.774522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:12.774850 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.774836 2576 scope.go:117] "RemoveContainer" containerID="0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda" Jul 08 02:44:12.775022 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:12.775002 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-786pp_openshift-console-operator(959b648d-524d-4fb4-961b-b32f222717ee)\"" pod="openshift-console-operator/console-operator-68b757fb67-786pp" podUID="959b648d-524d-4fb4-961b-b32f222717ee" Jul 08 02:44:12.849781 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.849681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:12.987432 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:12.987410 2576 scope.go:117] "RemoveContainer" containerID="0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda" Jul 08 02:44:12.987566 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:12.987550 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-786pp_openshift-console-operator(959b648d-524d-4fb4-961b-b32f222717ee)\"" pod="openshift-console-operator/console-operator-68b757fb67-786pp" podUID="959b648d-524d-4fb4-961b-b32f222717ee" Jul 08 02:44:13.056248 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:13.056225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:13.422731 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:13.422706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:14.048831 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.048800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:14.450906 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.450875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:14.658936 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.658910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:14.758213 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.758132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:14.758320 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.758229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:14.758391 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:14.758371 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:44:14.758440 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:14.758426 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls podName:ec513da9-d082-4b0a-a6fb-bb83cbb3ff51 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:22.758412607 +0000 UTC m=+253.036789786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjnfq" (UID: "ec513da9-d082-4b0a-a6fb-bb83cbb3ff51") : secret "insights-runtime-extractor-tls" not found Jul 08 02:44:14.758760 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.758742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:14.860880 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:14.860855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:15.256641 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:15.256612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:15.591199 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:15.591136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:15.854044 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:15.853982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:16.253434 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:16.253409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:16.316837 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:16.316814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:16.453605 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:16.453577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:16.648711 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:16.648681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:17.063973 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:17.063856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:17.252615 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:17.252586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:17.655186 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:17.655134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:18.182261 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.182233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:18.182261 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.182262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:18.182447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.182298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:18.182447 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:18.182426 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle podName:ccd97aff-9a22-4b34-84c2-3eb2ea9596a4 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:34.182407075 +0000 UTC m=+264.460784258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle") pod "router-default-5f8b685459-ftl64" (UID: "ccd97aff-9a22-4b34-84c2-3eb2ea9596a4") : configmap references non-existent config key: service-ca.crt Jul 08 02:44:18.184716 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.184680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-metrics-certs\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:18.184811 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.184724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5becb78c-aca9-4cd2-939b-3a5d94c1f9d2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jczdk\" (UID: \"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:18.254995 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.254967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:18.281475 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.281456 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2lb26\"" Jul 08 02:44:18.282548 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.282525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:18.284910 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.284892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"image-registry-7d66d495bb-fgs92\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:18.289725 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.289702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" Jul 08 02:44:18.395174 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.395149 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tthw\"" Jul 08 02:44:18.403233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.403198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:18.416138 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.416116 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk"] Jul 08 02:44:18.539470 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.539438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d66d495bb-fgs92"] Jul 08 02:44:18.546314 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:18.546280 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8f3b8d_2980_45e0_8994_4b7f0e3fa629.slice/crio-ec4ca5cebfb6af65e59e59763c75ed458b4c349cc99b3017cfbff5fa04168857 WatchSource:0}: Error finding container ec4ca5cebfb6af65e59e59763c75ed458b4c349cc99b3017cfbff5fa04168857: Status 404 returned error can't find the container with id ec4ca5cebfb6af65e59e59763c75ed458b4c349cc99b3017cfbff5fa04168857 Jul 08 02:44:18.648266 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.648246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:18.914459 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:18.914430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:19.009156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.009122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" event={"ID":"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629","Type":"ContainerStarted","Data":"4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc"} Jul 08 02:44:19.009343 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.009180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" event={"ID":"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629","Type":"ContainerStarted","Data":"ec4ca5cebfb6af65e59e59763c75ed458b4c349cc99b3017cfbff5fa04168857"} Jul 08 02:44:19.009343 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.009206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:19.010794 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.010769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" event={"ID":"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2","Type":"ContainerStarted","Data":"9c122f438c7bee2a624eac02ab975f826e8f4533732594b543bb1ec020f71a2a"} Jul 08 02:44:19.026947 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.026784 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" podStartSLOduration=17.026764689 podStartE2EDuration="17.026764689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:19.026706106 +0000 UTC m=+249.305083308" watchObservedRunningTime="2026-07-08 02:44:19.026764689 +0000 UTC m=+249.305141883" Jul 08 02:44:19.067445 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.067415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:19.251263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.251159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:19.652199 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:19.652167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:20.053982 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:20.053915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:20.851077 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:20.850999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:21.019373 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.019334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" event={"ID":"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2","Type":"ContainerStarted","Data":"8d78f4c030fb82cd9a9d725a905ef6dfd6c28ed7ebe7c296914e42ba2b3bfb53"} Jul 08 02:44:21.019503 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.019399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" event={"ID":"5becb78c-aca9-4cd2-939b-3a5d94c1f9d2","Type":"ContainerStarted","Data":"3529454d542dfb6ad5635667fe2b88a0a6f4301deafa08917abad442a6fb4150"} Jul 08 02:44:21.035556 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.035504 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jczdk" podStartSLOduration=16.974538231 podStartE2EDuration="19.035488642s" totalImagesPullingTime="2.060950411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:21.034167853 +0000 UTC m=+251.312545057" watchObservedRunningTime="2026-07-08 02:44:21.035488642 +0000 UTC m=+251.313865868" Jul 08 02:44:21.053914 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.053884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:21.457319 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.457289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:21.654101 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.654069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:21.911044 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.911002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:44:21.913474 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:21.913445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e669e30-a095-4fe7-be16-7ca17010a100-metrics-certs\") pod \"network-metrics-daemon-frgkz\" (UID: \"4e669e30-a095-4fe7-be16-7ca17010a100\") " pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:44:22.175458 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.175374 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n5djs\"" Jul 08 02:44:22.183195 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.183175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frgkz" Jul 08 02:44:22.253389 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.253336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:22.308432 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.308395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-frgkz"] Jul 08 02:44:22.313688 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:22.313662 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e669e30_a095_4fe7_be16_7ca17010a100.slice/crio-6bef93f001b17fc8ac8d936a6ac19f254328ffe3a5d9c4fa58cb21380a8c52dc WatchSource:0}: Error finding container 6bef93f001b17fc8ac8d936a6ac19f254328ffe3a5d9c4fa58cb21380a8c52dc: Status 404 returned error can't find the container with id 6bef93f001b17fc8ac8d936a6ac19f254328ffe3a5d9c4fa58cb21380a8c52dc Jul 08 02:44:22.658810 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.658778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:22.817777 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.817744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:22.820180 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.820159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec513da9-d082-4b0a-a6fb-bb83cbb3ff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjnfq\" (UID: \"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51\") " pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:22.856423 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.856384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:22.904333 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.904310 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6xd4b\"" Jul 08 02:44:22.912218 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:22.912162 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjnfq" Jul 08 02:44:23.026937 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:23.026901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frgkz" event={"ID":"4e669e30-a095-4fe7-be16-7ca17010a100","Type":"ContainerStarted","Data":"6bef93f001b17fc8ac8d936a6ac19f254328ffe3a5d9c4fa58cb21380a8c52dc"} Jul 08 02:44:23.044829 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:23.044791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjnfq"] Jul 08 02:44:23.051850 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:23.051821 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec513da9_d082_4b0a_a6fb_bb83cbb3ff51.slice/crio-67af47451a6238ea91e7feb6b2f76122a528b10abf0d678cccd6bf9b14b0fe3c WatchSource:0}: Error finding container 67af47451a6238ea91e7feb6b2f76122a528b10abf0d678cccd6bf9b14b0fe3c: Status 404 returned error can't find the container with id 67af47451a6238ea91e7feb6b2f76122a528b10abf0d678cccd6bf9b14b0fe3c Jul 08 02:44:23.060064 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:23.060038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:24.034892 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:24.034854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frgkz" event={"ID":"4e669e30-a095-4fe7-be16-7ca17010a100","Type":"ContainerStarted","Data":"4a44b41b409b7edcc7bd8934304801a2257bf31c4c21d1690c977b218b9d02a7"} Jul 08 02:44:24.035392 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:24.034922 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frgkz" event={"ID":"4e669e30-a095-4fe7-be16-7ca17010a100","Type":"ContainerStarted","Data":"b65637dcb3df675c853d2f059e1e7f682f04e95f202ee3e4be0a178bfc01dd74"} Jul 08 02:44:24.037555 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:24.037531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjnfq" event={"ID":"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51","Type":"ContainerStarted","Data":"67af47451a6238ea91e7feb6b2f76122a528b10abf0d678cccd6bf9b14b0fe3c"} Jul 08 02:44:24.051847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:24.051791 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-frgkz" podStartSLOduration=252.802212924 podStartE2EDuration="4m14.051772363s" totalImagesPullingTime="1.249559439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:24.051215956 +0000 UTC m=+254.329593158" watchObservedRunningTime="2026-07-08 02:44:24.051772363 +0000 UTC m=+254.330149566" Jul 08 02:44:26.273446 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:26.273407 2576 scope.go:117] "RemoveContainer" containerID="0252eb9a1e620f9e732d1d5d32c9502252bc9e43bb9dd3286681ba884ab1ddda" Jul 08 02:44:27.051158 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:27.051134 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:44:27.051300 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:27.051239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-786pp" event={"ID":"959b648d-524d-4fb4-961b-b32f222717ee","Type":"ContainerStarted","Data":"6185c86af064ea5ab683a14fd51f7dc429c4f954bfedf7e669695121baca9fd7"} Jul 08 02:44:27.051634 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:27.051579 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:27.052832 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:27.052808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjnfq" event={"ID":"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51","Type":"ContainerStarted","Data":"fe663ad8a98271fad29340bf9bdf44101ae33b45cfe696d1f3fccf6f4395178f"} Jul 08 02:44:27.067850 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:27.067807 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-786pp" podStartSLOduration=22.447631686 podStartE2EDuration="25.067790189s" totalImagesPullingTime="2.620158503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:27.065901916 +0000 UTC m=+257.344279117" watchObservedRunningTime="2026-07-08 02:44:27.067790189 +0000 UTC m=+257.346167392" Jul 08 02:44:28.051952 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.051909 2576 patch_prober.go:28] interesting pod/console-operator-68b757fb67-786pp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 02:44:28.052437 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.051976 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-786pp" podUID="959b648d-524d-4fb4-961b-b32f222717ee" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 08 02:44:28.059302 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.059265 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjnfq" event={"ID":"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51","Type":"ContainerStarted","Data":"f8db2d63e9f8962fdd1b216624b2ac546adf5c477e66bd0bee7e20f828072343"} Jul 08 02:44:28.284598 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.284564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:28.342371 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.342270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:28.664134 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.664101 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-786pp" Jul 08 02:44:28.848785 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.848748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:28.947134 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:28.947054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:29.143454 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:29.143423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:29.319341 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:29.319261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:29.382798 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:29.382764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:29.721028 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:29.720977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:29.776047 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:29.776011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:29.918655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:29.918622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:30.680216 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:30.680187 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d66d495bb-fgs92"] Jul 08 02:44:30.802500 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:30.802471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:30.899123 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:30.899089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:30.995895 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:30.995814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:31.071875 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:31.071839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjnfq" event={"ID":"ec513da9-d082-4b0a-a6fb-bb83cbb3ff51","Type":"ContainerStarted","Data":"359121febd5623fc6204f4d1f105d88be0ec5745114fddc6ee86240a4d97b070"} Jul 08 02:44:31.089432 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:31.089383 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hjnfq" podStartSLOduration=18.185826879 podStartE2EDuration="25.089345116s" totalImagesPullingTime="6.903518237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:44:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:31.088500875 +0000 UTC m=+261.366878077" watchObservedRunningTime="2026-07-08 02:44:31.089345116 +0000 UTC m=+261.367722317" Jul 08 02:44:31.149470 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:31.149439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:31.230563 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:31.230533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:34.209347 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:34.209316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:34.209862 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:34.209842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccd97aff-9a22-4b34-84c2-3eb2ea9596a4-service-ca-bundle\") pod \"router-default-5f8b685459-ftl64\" (UID: \"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4\") " pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:34.475456 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:34.475381 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tqcmf\"" Jul 08 02:44:34.483162 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:34.483134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:34.624447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:34.624414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f8b685459-ftl64"] Jul 08 02:44:34.627906 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:34.627883 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd97aff_9a22_4b34_84c2_3eb2ea9596a4.slice/crio-d67605a2ee3432b4689a80558150ebd93b5fd63cabbd306387da27e259f9ecd3 WatchSource:0}: Error finding container d67605a2ee3432b4689a80558150ebd93b5fd63cabbd306387da27e259f9ecd3: Status 404 returned error can't find the container with id d67605a2ee3432b4689a80558150ebd93b5fd63cabbd306387da27e259f9ecd3 Jul 08 02:44:35.086311 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:35.086267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f8b685459-ftl64" event={"ID":"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4","Type":"ContainerStarted","Data":"5f0e823543f43207613fccd9ebe3c50c7f75fea60ab4a083c2d4d1df79858fa5"} Jul 08 02:44:35.086505 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:35.086320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f8b685459-ftl64" event={"ID":"ccd97aff-9a22-4b34-84c2-3eb2ea9596a4","Type":"ContainerStarted","Data":"d67605a2ee3432b4689a80558150ebd93b5fd63cabbd306387da27e259f9ecd3"} Jul 08 02:44:35.104004 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:35.103957 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f8b685459-ftl64" podStartSLOduration=33.103926662 podStartE2EDuration="33.103926662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:35.103031499 +0000 UTC m=+265.381408712" watchObservedRunningTime="2026-07-08 02:44:35.103926662 +0000 UTC m=+265.382303864" Jul 08 02:44:35.483776 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:35.483755 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:35.486054 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:35.486034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:36.089881 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:36.089850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:36.090987 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:36.090967 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f8b685459-ftl64" Jul 08 02:44:38.245556 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.245529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:38.339603 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.339573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:38.405122 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.405089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:38.405294 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.405279 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx"] Jul 08 02:44:38.408129 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.408115 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:38.410611 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.410592 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q9lk2\"" Jul 08 02:44:38.410722 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.410651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 02:44:38.418166 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.418146 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx"] Jul 08 02:44:38.465962 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.465941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:38.539021 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.538964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/27ccdff5-d3ff-49e0-9643-b01f43e0946f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-p7nrx\" (UID: \"27ccdff5-d3ff-49e0-9643-b01f43e0946f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:38.639582 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.639552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/27ccdff5-d3ff-49e0-9643-b01f43e0946f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-p7nrx\" (UID: \"27ccdff5-d3ff-49e0-9643-b01f43e0946f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:38.641742 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.641722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/27ccdff5-d3ff-49e0-9643-b01f43e0946f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-p7nrx\" (UID: \"27ccdff5-d3ff-49e0-9643-b01f43e0946f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:38.716770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.716750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:38.841647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:38.841618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx"] Jul 08 02:44:38.845669 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:38.845637 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ccdff5_d3ff_49e0_9643_b01f43e0946f.slice/crio-ad93fbca1b0a9d6262301534e67f8438acb995dc5027c138fea4caf6957d33c0 WatchSource:0}: Error finding container ad93fbca1b0a9d6262301534e67f8438acb995dc5027c138fea4caf6957d33c0: Status 404 returned error can't find the container with id ad93fbca1b0a9d6262301534e67f8438acb995dc5027c138fea4caf6957d33c0 Jul 08 02:44:39.100155 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:39.100083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" event={"ID":"27ccdff5-d3ff-49e0-9643-b01f43e0946f","Type":"ContainerStarted","Data":"ad93fbca1b0a9d6262301534e67f8438acb995dc5027c138fea4caf6957d33c0"} Jul 08 02:44:39.746170 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:39.746124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:39.795138 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:39.795110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:40.685312 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:40.685269 2576 patch_prober.go:28] interesting pod/image-registry-7d66d495bb-fgs92 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 02:44:40.685521 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:40.685316 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 02:44:41.107778 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.107694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" event={"ID":"27ccdff5-d3ff-49e0-9643-b01f43e0946f","Type":"ContainerStarted","Data":"de4a456fb34246a999d021fd289636e23f3bd6e49df479cd08b4bf080c94e258"} Jul 08 02:44:41.108132 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.107920 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:41.112567 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.112543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" Jul 08 02:44:41.123332 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.123290 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-p7nrx" podStartSLOduration=1.8646861179999998 podStartE2EDuration="3.123276916s" totalImagesPullingTime="1.258590798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:41.122752767 +0000 UTC m=+271.401129971" watchObservedRunningTime="2026-07-08 02:44:41.123276916 +0000 UTC m=+271.401654118" Jul 08 02:44:41.430846 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.430809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:41.531339 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.531302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:41.657707 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.657667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:41.758553 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.758479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:41.810325 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:41.810303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:43.281842 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:43.281813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:43.353137 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:43.353105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:43.640219 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:43.640189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:44.551345 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:44.551299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:44.651584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:44.651552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:45.568811 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.568773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:45.741618 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.741588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:45.802954 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.802924 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd"] Jul 08 02:44:45.808496 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.808474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:45.808707 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.808691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:45.811206 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.811155 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 02:44:45.811206 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.811163 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 02:44:45.811206 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.811186 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 02:44:45.811450 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.811437 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 02:44:45.812278 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.812259 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x96vv\"" Jul 08 02:44:45.812413 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.812300 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 02:44:45.816591 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.816526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd"] Jul 08 02:44:45.829727 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.829675 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4dxkb"] Jul 08 02:44:45.833424 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.833405 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.835744 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.835727 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 02:44:45.835908 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.835789 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 02:44:45.835908 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.835892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 02:44:45.836091 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.836072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f7vjp\"" Jul 08 02:44:45.888489 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.888462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:45.997047 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:45.997236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-textfile\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ac39c34-7009-471e-832e-5c944fe3a1d0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:45.997236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qmf\" (UniqueName: \"kubernetes.io/projected/3ac39c34-7009-471e-832e-5c944fe3a1d0-kube-api-access-72qmf\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:45.997236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphft\" (UniqueName: \"kubernetes.io/projected/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-kube-api-access-bphft\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:45.997471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-metrics-client-ca\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-root\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997471 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997666 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-sys\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:45.997666 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:45.997556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-wtmp\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.087127 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.087053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:46.098520 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphft\" (UniqueName: \"kubernetes.io/projected/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-kube-api-access-bphft\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.098659 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.098659 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-metrics-client-ca\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.098762 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:46.098670 2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 08 02:44:46.098762 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:46.098738 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-tls podName:3ac39c34-7009-471e-832e-5c944fe3a1d0 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:46.598716119 +0000 UTC m=+276.877093303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-l5xdd" (UID: "3ac39c34-7009-471e-832e-5c944fe3a1d0") : secret "openshift-state-metrics-tls" not found Jul 08 02:44:46.098889 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.098889 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.098889 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-root\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.098889 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-sys\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-wtmp\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.098972 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-root\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-textfile\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ac39c34-7009-471e-832e-5c944fe3a1d0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.099147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qmf\" (UniqueName: \"kubernetes.io/projected/3ac39c34-7009-471e-832e-5c944fe3a1d0-kube-api-access-72qmf\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.099524 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-metrics-client-ca\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099524 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-sys\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099623 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099539 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-textfile\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099675 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:46.099635 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 02:44:46.099728 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:46.099684 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls podName:8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:46.599667407 +0000 UTC m=+276.878044630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls") pod "node-exporter-4dxkb" (UID: "8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0") : secret "node-exporter-tls" not found Jul 08 02:44:46.099728 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099679 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099826 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-wtmp\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.099971 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.099950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ac39c34-7009-471e-832e-5c944fe3a1d0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.101921 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.101898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.102403 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.102386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.107847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.107824 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphft\" (UniqueName: \"kubernetes.io/projected/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-kube-api-access-bphft\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.107942 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.107905 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qmf\" (UniqueName: \"kubernetes.io/projected/3ac39c34-7009-471e-832e-5c944fe3a1d0-kube-api-access-72qmf\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.197289 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.197254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:46.298842 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.298810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:46.603236 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.603204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.603608 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.603244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:46.603608 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:46.603348 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 02:44:46.603608 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:46.603424 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls podName:8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0 nodeName:}" failed. No retries permitted until 2026-07-08 02:44:47.603410293 +0000 UTC m=+277.881787473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls") pod "node-exporter-4dxkb" (UID: "8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0") : secret "node-exporter-tls" not found Jul 08 02:44:46.605861 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.605837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac39c34-7009-471e-832e-5c944fe3a1d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-l5xdd\" (UID: \"3ac39c34-7009-471e-832e-5c944fe3a1d0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.719383 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.719344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" Jul 08 02:44:46.788819 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.788778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:46.854146 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.854064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd"] Jul 08 02:44:46.856443 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.856423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:46.858243 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:46.858224 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac39c34_7009_471e_832e_5c944fe3a1d0.slice/crio-48a1570a57c3ac311d9083481b2cde7bf3e80ac7dc05b7a15c4af1ca7fbbd8b1 WatchSource:0}: Error finding container 48a1570a57c3ac311d9083481b2cde7bf3e80ac7dc05b7a15c4af1ca7fbbd8b1: Status 404 returned error can't find the container with id 48a1570a57c3ac311d9083481b2cde7bf3e80ac7dc05b7a15c4af1ca7fbbd8b1 Jul 08 02:44:46.916114 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.916095 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 02:44:46.920785 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.920769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:46.923194 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 02:44:46.923194 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923104 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 02:44:46.923194 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923129 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 02:44:46.923610 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 02:44:46.923610 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 02:44:46.923610 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 02:44:46.923835 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 02:44:46.923835 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.923798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m59s5\"" Jul 08 02:44:46.924015 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.924000 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 02:44:46.924077 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.924001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 02:44:46.940160 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:46.940139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 02:44:47.000212 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.000192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:47.061890 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.061856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:47.106858 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.106782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c0734c-c1cb-46cc-945d-3f7352a37a33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.106858 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.106826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.106886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-config-volume\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.106969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9ds\" (UniqueName: \"kubernetes.io/projected/96c0734c-c1cb-46cc-945d-3f7352a37a33-kube-api-access-dc9ds\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-web-config\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c0734c-c1cb-46cc-945d-3f7352a37a33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c0734c-c1cb-46cc-945d-3f7352a37a33-config-out\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c0734c-c1cb-46cc-945d-3f7352a37a33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.107315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.107306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c0734c-c1cb-46cc-945d-3f7352a37a33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.130064 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.130029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" event={"ID":"3ac39c34-7009-471e-832e-5c944fe3a1d0","Type":"ContainerStarted","Data":"4f4992b1955a29d26241dbd27e68ef8671a0acfcdeec19f50d73b25beee9a47b"} Jul 08 02:44:47.130296 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.130085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" event={"ID":"3ac39c34-7009-471e-832e-5c944fe3a1d0","Type":"ContainerStarted","Data":"6cca2ddbb571903a71a3df65fbcf920b539be593693514d169872310a9a6d63d"} Jul 08 02:44:47.130296 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.130107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" event={"ID":"3ac39c34-7009-471e-832e-5c944fe3a1d0","Type":"ContainerStarted","Data":"48a1570a57c3ac311d9083481b2cde7bf3e80ac7dc05b7a15c4af1ca7fbbd8b1"} Jul 08 02:44:47.208282 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c0734c-c1cb-46cc-945d-3f7352a37a33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.208282 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.208513 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-config-volume\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.208513 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.208513 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.208513 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9ds\" (UniqueName: \"kubernetes.io/projected/96c0734c-c1cb-46cc-945d-3f7352a37a33-kube-api-access-dc9ds\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.208513 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-web-config\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.209128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c0734c-c1cb-46cc-945d-3f7352a37a33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.209128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c0734c-c1cb-46cc-945d-3f7352a37a33-config-out\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.209128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c0734c-c1cb-46cc-945d-3f7352a37a33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.209128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.208961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.209128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.209013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.209128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.209038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c0734c-c1cb-46cc-945d-3f7352a37a33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.210167 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.209754 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c0734c-c1cb-46cc-945d-3f7352a37a33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.210167 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.209889 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c0734c-c1cb-46cc-945d-3f7352a37a33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.210338 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.210295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c0734c-c1cb-46cc-945d-3f7352a37a33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.211786 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.211760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-web-config\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.212041 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.212019 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.212312 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.212249 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-config-volume\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.212440 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.212399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.212531 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.212503 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.212737 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.212717 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c0734c-c1cb-46cc-945d-3f7352a37a33-config-out\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.212917 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.212897 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.213088 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.213071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c0734c-c1cb-46cc-945d-3f7352a37a33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.213425 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.213409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c0734c-c1cb-46cc-945d-3f7352a37a33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.216147 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.216129 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9ds\" (UniqueName: \"kubernetes.io/projected/96c0734c-c1cb-46cc-945d-3f7352a37a33-kube-api-access-dc9ds\") pod \"alertmanager-main-0\" (UID: \"96c0734c-c1cb-46cc-945d-3f7352a37a33\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.232006 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.231988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:44:47.375032 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.374933 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 02:44:47.378279 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:47.378253 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c0734c_c1cb_46cc_945d_3f7352a37a33.slice/crio-aa127060802195bbdd7af4902206e1859e14acc3a3b3b8ded37fa8f162f6921d WatchSource:0}: Error finding container aa127060802195bbdd7af4902206e1859e14acc3a3b3b8ded37fa8f162f6921d: Status 404 returned error can't find the container with id aa127060802195bbdd7af4902206e1859e14acc3a3b3b8ded37fa8f162f6921d Jul 08 02:44:47.612893 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.612857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:47.615414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.615393 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0-node-exporter-tls\") pod \"node-exporter-4dxkb\" (UID: \"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0\") " pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:47.642457 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.642393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4dxkb" Jul 08 02:44:47.655113 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:47.655066 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea83bf1_92b3_4f9a_836b_d3c07dc09cd0.slice/crio-4438ba2945a7661f76ef26ced4504c0331757c1443171cb34284cf3a1b08cc42 WatchSource:0}: Error finding container 4438ba2945a7661f76ef26ced4504c0331757c1443171cb34284cf3a1b08cc42: Status 404 returned error can't find the container with id 4438ba2945a7661f76ef26ced4504c0331757c1443171cb34284cf3a1b08cc42 Jul 08 02:44:47.796565 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.796530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:47.854380 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:47.854341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:48.137913 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.137876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" event={"ID":"3ac39c34-7009-471e-832e-5c944fe3a1d0","Type":"ContainerStarted","Data":"091a836d914bffeaa984a1d5277c32e7b8fcfb9187add96130c6f1250a058cdb"} Jul 08 02:44:48.138956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.138934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"aa127060802195bbdd7af4902206e1859e14acc3a3b3b8ded37fa8f162f6921d"} Jul 08 02:44:48.140023 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.140004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dxkb" event={"ID":"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0","Type":"ContainerStarted","Data":"4438ba2945a7661f76ef26ced4504c0331757c1443171cb34284cf3a1b08cc42"} Jul 08 02:44:48.154897 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.154856 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-l5xdd" podStartSLOduration=2.158309834 podStartE2EDuration="3.154843789s" totalImagesPullingTime="996.533955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:48.154118664 +0000 UTC m=+278.432495878" watchObservedRunningTime="2026-07-08 02:44:48.154843789 +0000 UTC m=+278.433220990" Jul 08 02:44:48.735343 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.735298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:48.835656 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.835607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:48.936004 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:48.935971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:49.146203 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:49.146175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dxkb" event={"ID":"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0","Type":"ContainerStarted","Data":"4cff805f0aad020b0934016a72069dd7d84da4835c77a4f656d4d3ad6942a80b"} Jul 08 02:44:49.147762 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:49.147740 2576 generic.go:356] "Generic (PLEG): container finished" podID="96c0734c-c1cb-46cc-945d-3f7352a37a33" containerID="daafa5514774d62588239b7ee2374ac3881d4c9b26159f21df7a9eee1ec5595e" exitCode=0 Jul 08 02:44:49.147848 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:49.147819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerDied","Data":"daafa5514774d62588239b7ee2374ac3881d4c9b26159f21df7a9eee1ec5595e"} Jul 08 02:44:49.974650 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:49.974617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:50.152640 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.152608 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0" containerID="4cff805f0aad020b0934016a72069dd7d84da4835c77a4f656d4d3ad6942a80b" exitCode=0 Jul 08 02:44:50.152640 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.152643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dxkb" event={"ID":"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0","Type":"ContainerDied","Data":"4cff805f0aad020b0934016a72069dd7d84da4835c77a4f656d4d3ad6942a80b"} Jul 08 02:44:50.606413 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.606298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:50.667016 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.666982 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-855974978b-nclkt"] Jul 08 02:44:50.671847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.671725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.674532 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.674502 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 02:44:50.674653 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.674616 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 02:44:50.674768 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.674750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 02:44:50.675447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.675426 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 02:44:50.675557 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.675487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 02:44:50.675696 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.675676 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 02:44:50.675776 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.675722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ws27b\"" Jul 08 02:44:50.675776 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.675749 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 02:44:50.681469 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.680885 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 02:44:50.681469 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.681231 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855974978b-nclkt"] Jul 08 02:44:50.686113 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.686077 2576 patch_prober.go:28] interesting pod/image-registry-7d66d495bb-fgs92 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 02:44:50.686218 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.686130 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 02:44:50.708223 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.708200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:50.742956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.742930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-config\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.743068 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.742963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-oauth-serving-cert\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.743068 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.742987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-oauth-config\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.743068 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.743052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6xl\" (UniqueName: \"kubernetes.io/projected/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-kube-api-access-qj6xl\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.743223 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.743073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-service-ca\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.743223 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.743099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-serving-cert\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.743304 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.743217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-trusted-ca-bundle\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844505 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-serving-cert\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844622 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-trusted-ca-bundle\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844622 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-config\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844622 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-oauth-serving-cert\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844795 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-oauth-config\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844795 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6xl\" (UniqueName: \"kubernetes.io/projected/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-kube-api-access-qj6xl\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.844964 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.844939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-service-ca\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.845450 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.845395 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-oauth-serving-cert\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.845450 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.845395 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-config\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.845612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.845560 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-trusted-ca-bundle\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.845732 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.845670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-service-ca\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.847912 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.847475 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-serving-cert\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.847912 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.847482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-oauth-config\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.853209 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.853189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6xl\" (UniqueName: \"kubernetes.io/projected/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-kube-api-access-qj6xl\") pod \"console-855974978b-nclkt\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:50.984368 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:50.984330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:44:51.135322 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.135297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855974978b-nclkt"] Jul 08 02:44:51.138907 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:51.138884 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2b529e_41ac_4e9b_96bb_8ad7e50031d0.slice/crio-b31d00dbebd8f905978a9194961827b10da9368f74e54f43d56fe5affc3827b1 WatchSource:0}: Error finding container b31d00dbebd8f905978a9194961827b10da9368f74e54f43d56fe5affc3827b1: Status 404 returned error can't find the container with id b31d00dbebd8f905978a9194961827b10da9368f74e54f43d56fe5affc3827b1 Jul 08 02:44:51.160068 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.160043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"d24a31df6d9d0bc5d68d02a5ab71bcd850e535d456acd28c8e903f7008904fed"} Jul 08 02:44:51.160176 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.160122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"da4515c71c870f4270d157173260958e5be1971e092c9b5c66f7ee7f455ef1fa"} Jul 08 02:44:51.160176 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.160167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"d3a5553256ec61307d512c636f7583f9581f94c36ff9efa6e4184193ed58e7f0"} Jul 08 02:44:51.160292 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.160204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"beab306b489eb636c63a323555cfb8fccd8538344f47ffb4838accb7ec8221f5"} Jul 08 02:44:51.160292 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.160247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"1187fc8cb8ac5f1d2258b9556d405666f098cb6292619e340ba4871d2dd7c857"} Jul 08 02:44:51.162164 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.162143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dxkb" event={"ID":"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0","Type":"ContainerStarted","Data":"bcf0ffb68474cccf2e72f7a66df4e25ebee94a4d503705dac19cb8e7e10c132a"} Jul 08 02:44:51.162264 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.162182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dxkb" event={"ID":"8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0","Type":"ContainerStarted","Data":"2231a308896939b71c40ff5a5ee0a5617d482e8a6cd9bb4e606ebf94d3f62876"} Jul 08 02:44:51.163322 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.163300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855974978b-nclkt" event={"ID":"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0","Type":"ContainerStarted","Data":"b31d00dbebd8f905978a9194961827b10da9368f74e54f43d56fe5affc3827b1"} Jul 08 02:44:51.178895 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.178666 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4dxkb" podStartSLOduration=4.736874046 podStartE2EDuration="6.178649772s" totalImagesPullingTime="1.367514035s" totalInitContainerRuntime="74.261691ms" isStatefulPod=true podCreationTimestamp="2026-07-08 02:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:51.178060804 +0000 UTC m=+281.456438007" watchObservedRunningTime="2026-07-08 02:44:51.178649772 +0000 UTC m=+281.457026977" Jul 08 02:44:51.280792 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:51.280769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:51.765118 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:51.765057 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tdxfs" podUID="a6878310-383d-46ab-b566-005f4981c3f7" Jul 08 02:44:52.008368 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:52.008316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:52.172782 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:52.172535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdxfs" Jul 08 02:44:52.172782 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:52.172562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c0734c-c1cb-46cc-945d-3f7352a37a33","Type":"ContainerStarted","Data":"8470e6513b75f9073c86dd295cfbd2f2a2200282ab0fb2447c124485fe264ab1"} Jul 08 02:44:52.198770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:52.198713 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.163042491 podStartE2EDuration="6.198684263s" totalImagesPullingTime="4.019547094s" totalInitContainerRuntime="16.094678ms" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:52.197483077 +0000 UTC m=+282.475860280" watchObservedRunningTime="2026-07-08 02:44:52.198684263 +0000 UTC m=+282.477061482" Jul 08 02:44:54.181649 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:54.181618 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855974978b-nclkt" event={"ID":"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0","Type":"ContainerStarted","Data":"6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f"} Jul 08 02:44:54.197562 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:54.197513 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-855974978b-nclkt" podStartSLOduration=1.495479114 podStartE2EDuration="4.197497525s" totalImagesPullingTime="2.702018411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:54.196728577 +0000 UTC m=+284.475105792" watchObservedRunningTime="2026-07-08 02:44:54.197497525 +0000 UTC m=+284.475874726" Jul 08 02:44:55.485266 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.485221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:44:55.487626 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.487602 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6878310-383d-46ab-b566-005f4981c3f7-metrics-tls\") pod \"dns-default-tdxfs\" (UID: \"a6878310-383d-46ab-b566-005f4981c3f7\") " pod="openshift-dns/dns-default-tdxfs" Jul 08 02:44:55.500600 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.500566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:55.574649 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.574627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:55.699845 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.699816 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerName="registry" containerID="cri-o://4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc" gracePeriod=30 Jul 08 02:44:55.775942 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.775890 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8d2rg\"" Jul 08 02:44:55.784635 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.784609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdxfs" Jul 08 02:44:55.937195 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.937171 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:55.938794 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.938773 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tdxfs"] Jul 08 02:44:55.942879 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:44:55.942855 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6878310_383d_46ab_b566_005f4981c3f7.slice/crio-cbc281005ec41fc057b6e7d543fb6019aa0102905011abcfffa7190a0da3f8fa WatchSource:0}: Error finding container cbc281005ec41fc057b6e7d543fb6019aa0102905011abcfffa7190a0da3f8fa: Status 404 returned error can't find the container with id cbc281005ec41fc057b6e7d543fb6019aa0102905011abcfffa7190a0da3f8fa Jul 08 02:44:55.988305 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988267 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988427 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988337 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-installation-pull-secrets\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988427 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988373 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t8mz\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-kube-api-access-5t8mz\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988427 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988389 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-certificates\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988427 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988406 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-bound-sa-token\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988427 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-trusted-ca\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988660 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988478 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-ca-trust-extracted\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.988660 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988517 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-image-registry-private-configuration\") pod \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\" (UID: \"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629\") " Jul 08 02:44:55.989156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.988846 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:44:55.989156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.989056 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:44:55.990742 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.990708 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:44:55.991103 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.991072 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:44:55.991191 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.991115 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-kube-api-access-5t8mz" (OuterVolumeSpecName: "kube-api-access-5t8mz") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "kube-api-access-5t8mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:44:55.991191 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.991173 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:44:55.991275 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.991190 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:44:55.996463 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:55.996442 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" (UID: "4f8f3b8d-2980-45e0-8994-4b7f0e3fa629"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:44:56.089530 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089473 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-ca-trust-extracted\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089530 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089498 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-image-registry-private-configuration\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089530 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089514 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-tls\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089530 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089527 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-installation-pull-secrets\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089702 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089541 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8mz\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-kube-api-access-5t8mz\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089702 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089552 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-registry-certificates\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089702 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089564 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-bound-sa-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.089702 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.089575 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629-trusted-ca\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:44:56.189341 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.189311 2576 generic.go:356] "Generic (PLEG): container finished" podID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerID="4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc" exitCode=0 Jul 08 02:44:56.189452 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.189381 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" Jul 08 02:44:56.189452 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.189404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" event={"ID":"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629","Type":"ContainerDied","Data":"4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc"} Jul 08 02:44:56.189551 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.189459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66d495bb-fgs92" event={"ID":"4f8f3b8d-2980-45e0-8994-4b7f0e3fa629","Type":"ContainerDied","Data":"ec4ca5cebfb6af65e59e59763c75ed458b4c349cc99b3017cfbff5fa04168857"} Jul 08 02:44:56.189551 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.189489 2576 scope.go:117] "RemoveContainer" containerID="4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc" Jul 08 02:44:56.190773 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.190734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdxfs" event={"ID":"a6878310-383d-46ab-b566-005f4981c3f7","Type":"ContainerStarted","Data":"cbc281005ec41fc057b6e7d543fb6019aa0102905011abcfffa7190a0da3f8fa"} Jul 08 02:44:56.198475 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.198461 2576 scope.go:117] "RemoveContainer" containerID="4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc" Jul 08 02:44:56.198715 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:44:56.198696 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc\": container with ID starting with 4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc not found: ID does not exist" containerID="4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc" Jul 08 02:44:56.198765 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.198724 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc"} err="failed to get container status \"4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc\": rpc error: code = NotFound desc = could not find container \"4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc\": container with ID starting with 4b1108022432a8149d0923d060b264c531f3cbdc27f95c50923321f235729edc not found: ID does not exist" Jul 08 02:44:56.210821 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.210801 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d66d495bb-fgs92"] Jul 08 02:44:56.216440 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.216415 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d66d495bb-fgs92"] Jul 08 02:44:56.276742 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.276722 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" path="/var/lib/kubelet/pods/4f8f3b8d-2980-45e0-8994-4b7f0e3fa629/volumes" Jul 08 02:44:56.828964 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.828859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:56.924632 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:56.924601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:58.201408 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:58.201385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdxfs" event={"ID":"a6878310-383d-46ab-b566-005f4981c3f7","Type":"ContainerStarted","Data":"e9b65963adeb34cc165ff92a7de8ea919df4b6aec63c00dc285b5415efba26a5"} Jul 08 02:44:58.282397 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:58.282347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:58.382971 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:58.382946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:44:59.206764 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:59.206734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdxfs" event={"ID":"a6878310-383d-46ab-b566-005f4981c3f7","Type":"ContainerStarted","Data":"58517353afb717fab4a20fecd16b9625c54e892f10fcf38c6c3ca3f74c2935fc"} Jul 08 02:44:59.207199 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:59.206856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tdxfs" Jul 08 02:44:59.221696 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:44:59.221653 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tdxfs" podStartSLOduration=252.103326036 podStartE2EDuration="4m14.221638974s" totalImagesPullingTime="2.118312938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:44:59.221545648 +0000 UTC m=+289.499922850" watchObservedRunningTime="2026-07-08 02:44:59.221638974 +0000 UTC m=+289.500016195" Jul 08 02:45:00.984741 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:00.984704 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:45:00.985099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:00.984764 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:45:00.985973 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:00.985954 2576 patch_prober.go:28] interesting pod/console-855974978b-nclkt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body= Jul 08 02:45:00.986019 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:00.985993 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-855974978b-nclkt" podUID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" containerName="console" probeResult="failure" output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused" Jul 08 02:45:05.189574 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:05.189543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:06.584516 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:06.584473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:07.847633 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:07.847601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:07.896449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:07.896423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:09.213475 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:09.213447 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tdxfs" Jul 08 02:45:10.110474 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:10.110451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:45:10.110721 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:10.110707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:45:10.121630 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:10.121615 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 02:45:10.988749 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:10.988722 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:45:10.992609 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:10.992593 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:45:11.153852 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:11.153820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:11.256099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:11.256032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:11.306419 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:11.306397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:14.877182 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:14.877147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:14.927530 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:14.927497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:15.769641 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:15.769608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:15.869733 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:15.869704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:15.985864 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:15.985832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:16.086274 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:16.086191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:21.231762 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:21.231732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:21.290569 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:21.290539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:21.393173 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:21.393145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:21.441864 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:21.441844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:23.887604 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:23.887573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:23.942045 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:23.942017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:24.782074 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:24.782044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:24.883558 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:24.883528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:26.307583 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:26.307552 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b1e0477-6102-4d78-8329-f3792ea19ef5" containerID="256de28a5e88666ba65e6faa410228b226566bae9b2bece03d7fa1681feb4289" exitCode=0 Jul 08 02:45:26.307912 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:26.307620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" event={"ID":"7b1e0477-6102-4d78-8329-f3792ea19ef5","Type":"ContainerDied","Data":"256de28a5e88666ba65e6faa410228b226566bae9b2bece03d7fa1681feb4289"} Jul 08 02:45:26.307957 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:26.307941 2576 scope.go:117] "RemoveContainer" containerID="256de28a5e88666ba65e6faa410228b226566bae9b2bece03d7fa1681feb4289" Jul 08 02:45:27.314849 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:27.314817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wphwj" event={"ID":"7b1e0477-6102-4d78-8329-f3792ea19ef5","Type":"ContainerStarted","Data":"1b522ece5853291de91286aa36764a98b6015c23157912f6f5a5cccd31b5dba6"} Jul 08 02:45:28.282971 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:28.282940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:28.383176 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:28.383142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:31.068132 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:31.068096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:32.977895 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:32.977863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:33.031628 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:33.031601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:37.889109 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:37.889076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:37.961541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:37.961511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:43.287839 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:43.287807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:43.348623 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:43.348595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:50.494038 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:50.494005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:50.594858 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:50.594832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:50.656941 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:50.656916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:58.284812 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:58.284770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:58.348335 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:58.348277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:58.985192 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:58.985160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:45:59.043914 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:45:59.043885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:05.076195 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:05.076125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:05.584330 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:05.584305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:10.179543 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.179510 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9bc669fb7-grtwd"] Jul 08 02:46:10.179878 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.179795 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerName="registry" Jul 08 02:46:10.179878 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.179806 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerName="registry" Jul 08 02:46:10.179878 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.179849 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8f3b8d-2980-45e0-8994-4b7f0e3fa629" containerName="registry" Jul 08 02:46:10.181908 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.181891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.184322 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.184284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 08 02:46:10.184322 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.184298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zqx9r\"" Jul 08 02:46:10.184519 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.184371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 08 02:46:10.184519 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.184294 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 08 02:46:10.184731 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.184715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 08 02:46:10.184787 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.184761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 08 02:46:10.189631 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.189613 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 08 02:46:10.198119 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.198101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9bc669fb7-grtwd"] Jul 08 02:46:10.252944 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.252915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253104 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.252975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-metrics-client-ca\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253104 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.253031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-serving-certs-ca-bundle\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253104 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.253072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253104 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.253098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-federate-client-tls\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253235 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.253124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8w66\" (UniqueName: \"kubernetes.io/projected/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-kube-api-access-x8w66\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253235 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.253189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-secret-telemeter-client\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.253235 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.253205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-telemeter-client-tls\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.353699 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.353654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-secret-telemeter-client\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.353699 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.353702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-telemeter-client-tls\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354044 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.353738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354044 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.353794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-metrics-client-ca\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354044 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.353995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-serving-certs-ca-bundle\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354044 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.354040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354257 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.354071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-federate-client-tls\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354257 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.354108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8w66\" (UniqueName: \"kubernetes.io/projected/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-kube-api-access-x8w66\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354750 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.354728 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.354900 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.354880 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-serving-certs-ca-bundle\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.355335 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.355314 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-metrics-client-ca\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.356463 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.356435 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-telemeter-client-tls\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.356550 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.356482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-secret-telemeter-client\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.356820 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.356804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.356981 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.356961 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-federate-client-tls\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.362549 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.362532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8w66\" (UniqueName: \"kubernetes.io/projected/f4ec95c0-1f4f-4666-935b-e3104fa2bafe-kube-api-access-x8w66\") pod \"telemeter-client-9bc669fb7-grtwd\" (UID: \"f4ec95c0-1f4f-4666-935b-e3104fa2bafe\") " pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.491230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.491155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" Jul 08 02:46:10.631125 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.631093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9bc669fb7-grtwd"] Jul 08 02:46:10.635053 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:46:10.635031 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ec95c0_1f4f_4666_935b_e3104fa2bafe.slice/crio-2e2618c1a3c6be0a17f4fd3a226c0732c08ed9deaafff9778541aea28dda814b WatchSource:0}: Error finding container 2e2618c1a3c6be0a17f4fd3a226c0732c08ed9deaafff9778541aea28dda814b: Status 404 returned error can't find the container with id 2e2618c1a3c6be0a17f4fd3a226c0732c08ed9deaafff9778541aea28dda814b Jul 08 02:46:10.637346 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:10.637332 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:46:11.472959 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:11.472920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" event={"ID":"f4ec95c0-1f4f-4666-935b-e3104fa2bafe","Type":"ContainerStarted","Data":"2e2618c1a3c6be0a17f4fd3a226c0732c08ed9deaafff9778541aea28dda814b"} Jul 08 02:46:13.290849 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:13.290775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:13.365384 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:13.365330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:13.482612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:13.482577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" event={"ID":"f4ec95c0-1f4f-4666-935b-e3104fa2bafe","Type":"ContainerStarted","Data":"7b79b9c187def7a212b4663bdaa3f6602dd4a72d59d464922760cea101382162"} Jul 08 02:46:13.482791 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:13.482632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" event={"ID":"f4ec95c0-1f4f-4666-935b-e3104fa2bafe","Type":"ContainerStarted","Data":"82a68e0e5360abd81f8908e5d72f62513cae81fe01460aa4e6c6f64573b5aa34"} Jul 08 02:46:13.482791 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:13.482662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" event={"ID":"f4ec95c0-1f4f-4666-935b-e3104fa2bafe","Type":"ContainerStarted","Data":"a31359b8c75ace3f0683e2c79ae499381e33de1ca40c34bd3375cacb245eb926"} Jul 08 02:46:13.506909 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:13.506861 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9bc669fb7-grtwd" podStartSLOduration=1.227113629 podStartE2EDuration="3.506846243s" totalImagesPullingTime="2.279732614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:46:13.505333304 +0000 UTC m=+363.783710506" watchObservedRunningTime="2026-07-08 02:46:13.506846243 +0000 UTC m=+363.785223444" Jul 08 02:46:14.467912 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:14.467881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:14.523002 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:14.522973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:15.123742 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:15.123712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:15.175049 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:15.175018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:20.316086 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:20.316053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:20.379280 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:20.379243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:24.917458 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:24.917419 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-855974978b-nclkt"] Jul 08 02:46:25.026595 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.026562 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c66fb6bb6-57vt4"] Jul 08 02:46:25.029106 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.029085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.044667 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.044641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c66fb6bb6-57vt4"] Jul 08 02:46:25.093609 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.093581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:25.171884 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.171804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:25.176046 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-oauth-serving-cert\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.176121 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-console-config\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.176167 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-oauth-config\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.176167 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-serving-cert\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.176265 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-trusted-ca-bundle\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.176313 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kjb\" (UniqueName: \"kubernetes.io/projected/d8946432-a817-48c2-a17e-3ebed7649536-kube-api-access-x5kjb\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.176313 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.176289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-service-ca\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277479 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-oauth-config\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-serving-cert\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-trusted-ca-bundle\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kjb\" (UniqueName: \"kubernetes.io/projected/d8946432-a817-48c2-a17e-3ebed7649536-kube-api-access-x5kjb\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-service-ca\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277655 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-oauth-serving-cert\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.277914 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.277659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-console-config\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.278448 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.278423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-service-ca\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.278548 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.278447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-oauth-serving-cert\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.278548 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.278425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-console-config\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.278870 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.278849 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-trusted-ca-bundle\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.280404 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.280384 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-serving-cert\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.280458 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.280384 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-oauth-config\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.287868 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.287844 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kjb\" (UniqueName: \"kubernetes.io/projected/d8946432-a817-48c2-a17e-3ebed7649536-kube-api-access-x5kjb\") pod \"console-7c66fb6bb6-57vt4\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.338708 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.338680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:25.383219 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.383181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:25.463743 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.463675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:25.479813 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.479794 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c66fb6bb6-57vt4"] Jul 08 02:46:25.482448 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:46:25.482422 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8946432_a817_48c2_a17e_3ebed7649536.slice/crio-b0eb3405bb686fe98d03925ed49613bceee906df9311e8b21b023b066867839b WatchSource:0}: Error finding container b0eb3405bb686fe98d03925ed49613bceee906df9311e8b21b023b066867839b: Status 404 returned error can't find the container with id b0eb3405bb686fe98d03925ed49613bceee906df9311e8b21b023b066867839b Jul 08 02:46:25.531189 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:25.531159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c66fb6bb6-57vt4" event={"ID":"d8946432-a817-48c2-a17e-3ebed7649536","Type":"ContainerStarted","Data":"b0eb3405bb686fe98d03925ed49613bceee906df9311e8b21b023b066867839b"} Jul 08 02:46:26.536536 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:26.536502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c66fb6bb6-57vt4" event={"ID":"d8946432-a817-48c2-a17e-3ebed7649536","Type":"ContainerStarted","Data":"e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d"} Jul 08 02:46:26.606710 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:26.606657 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c66fb6bb6-57vt4" podStartSLOduration=2.606639837 podStartE2EDuration="2.606639837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:46:26.580154059 +0000 UTC m=+376.858531262" watchObservedRunningTime="2026-07-08 02:46:26.606639837 +0000 UTC m=+376.885017099" Jul 08 02:46:35.339846 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.339799 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:35.340233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.339874 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:35.344368 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.344338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:35.574691 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.574663 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:46:35.607647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.607559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:35.707686 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.707659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:35.809163 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.809133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:35.908733 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:35.908703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:43.360721 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:43.360665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:43.480398 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:43.480365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:46:49.940308 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:49.940238 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-855974978b-nclkt" podUID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" containerName="console" containerID="cri-o://6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f" gracePeriod=15 Jul 08 02:46:50.173718 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.173690 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:46:50.175770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.175754 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj6xl\" (UniqueName: \"kubernetes.io/projected/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-kube-api-access-qj6xl\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.175825 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.175782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-trusted-ca-bundle\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.175825 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.175802 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-serving-cert\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.175897 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.175828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-oauth-config\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.176002 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.175967 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-config\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.176118 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176029 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-oauth-serving-cert\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.176118 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176061 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-service-ca\") pod \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\" (UID: \"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0\") " Jul 08 02:46:50.176242 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176155 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:46:50.176504 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176416 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-config" (OuterVolumeSpecName: "console-config") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:46:50.176504 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176454 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:46:50.176800 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176483 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-trusted-ca-bundle\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.176800 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.176694 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:46:50.178124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.178101 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-kube-api-access-qj6xl" (OuterVolumeSpecName: "kube-api-access-qj6xl") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "kube-api-access-qj6xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:46:50.178124 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.178107 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:46:50.178233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.178143 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" (UID: "6b2b529e-41ac-4e9b-96bb-8ad7e50031d0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:46:50.277050 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.276981 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6xl\" (UniqueName: \"kubernetes.io/projected/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-kube-api-access-qj6xl\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.277050 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.277006 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-serving-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.277050 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.277018 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-oauth-config\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.277050 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.277030 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-console-config\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.277050 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.277042 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-oauth-serving-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.277317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.277055 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0-service-ca\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:46:50.627458 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.627427 2576 generic.go:356] "Generic (PLEG): container finished" podID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" containerID="6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f" exitCode=0 Jul 08 02:46:50.627573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.627467 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855974978b-nclkt" Jul 08 02:46:50.627573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.627508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855974978b-nclkt" event={"ID":"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0","Type":"ContainerDied","Data":"6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f"} Jul 08 02:46:50.627573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.627550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855974978b-nclkt" event={"ID":"6b2b529e-41ac-4e9b-96bb-8ad7e50031d0","Type":"ContainerDied","Data":"b31d00dbebd8f905978a9194961827b10da9368f74e54f43d56fe5affc3827b1"} Jul 08 02:46:50.627573 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.627569 2576 scope.go:117] "RemoveContainer" containerID="6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f" Jul 08 02:46:50.635795 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.635780 2576 scope.go:117] "RemoveContainer" containerID="6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f" Jul 08 02:46:50.638301 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:46:50.636508 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f\": container with ID starting with 6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f not found: ID does not exist" containerID="6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f" Jul 08 02:46:50.638301 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.636541 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f"} err="failed to get container status \"6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f\": rpc error: code = NotFound desc = could not find container \"6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f\": container with ID starting with 6cf579d6a216229d2f4fbee2d4b6e65ba9ee7eef4cbe2717f47bbaf8fcb6497f not found: ID does not exist" Jul 08 02:46:50.651241 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.651218 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-855974978b-nclkt"] Jul 08 02:46:50.653274 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:50.653256 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-855974978b-nclkt"] Jul 08 02:46:52.277015 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:46:52.276979 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" path="/var/lib/kubelet/pods/6b2b529e-41ac-4e9b-96bb-8ad7e50031d0/volumes" Jul 08 02:47:09.638287 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:09.638259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:09.753709 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:09.753672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:11.153770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.153738 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl"] Jul 08 02:47:11.154127 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.154017 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" containerName="console" Jul 08 02:47:11.154127 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.154028 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" containerName="console" Jul 08 02:47:11.154127 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.154079 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2b529e-41ac-4e9b-96bb-8ad7e50031d0" containerName="console" Jul 08 02:47:11.156663 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.156647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.160130 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.160110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 02:47:11.161018 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.161002 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8mtf\"" Jul 08 02:47:11.161841 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.161821 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 02:47:11.175506 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.175479 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl"] Jul 08 02:47:11.238478 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.238448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.238640 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.238503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.238640 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.238560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rdx\" (UniqueName: \"kubernetes.io/projected/28c098dc-9c05-4116-aaeb-2e5377a4b128-kube-api-access-m4rdx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.339818 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.339780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.339946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.339835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.339946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.339855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rdx\" (UniqueName: \"kubernetes.io/projected/28c098dc-9c05-4116-aaeb-2e5377a4b128-kube-api-access-m4rdx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.340170 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.340152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.340206 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.340173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.360737 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.360701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rdx\" (UniqueName: \"kubernetes.io/projected/28c098dc-9c05-4116-aaeb-2e5377a4b128-kube-api-access-m4rdx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.465065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.464980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:11.611965 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.611930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl"] Jul 08 02:47:11.614240 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:47:11.614216 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c098dc_9c05_4116_aaeb_2e5377a4b128.slice/crio-c28000eeaf704dff723a9a6a6ee2cb847dc966e180d35a7ce6fd9911aa466bad WatchSource:0}: Error finding container c28000eeaf704dff723a9a6a6ee2cb847dc966e180d35a7ce6fd9911aa466bad: Status 404 returned error can't find the container with id c28000eeaf704dff723a9a6a6ee2cb847dc966e180d35a7ce6fd9911aa466bad Jul 08 02:47:11.714393 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:11.714343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" event={"ID":"28c098dc-9c05-4116-aaeb-2e5377a4b128","Type":"ContainerStarted","Data":"c28000eeaf704dff723a9a6a6ee2cb847dc966e180d35a7ce6fd9911aa466bad"} Jul 08 02:47:18.746933 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:18.746899 2576 generic.go:356] "Generic (PLEG): container finished" podID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerID="7c081495e916697a5bc1c2d956b48b8edde3e3fe514bb48491ddadd9743eb7cc" exitCode=0 Jul 08 02:47:18.747301 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:18.746986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" event={"ID":"28c098dc-9c05-4116-aaeb-2e5377a4b128","Type":"ContainerDied","Data":"7c081495e916697a5bc1c2d956b48b8edde3e3fe514bb48491ddadd9743eb7cc"} Jul 08 02:47:21.761141 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:21.761108 2576 generic.go:356] "Generic (PLEG): container finished" podID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerID="0b2ab57c9963092e388609035a78c9a5f76acd9fc7a2b172a5d20e7777c83682" exitCode=0 Jul 08 02:47:21.761531 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:21.761165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" event={"ID":"28c098dc-9c05-4116-aaeb-2e5377a4b128","Type":"ContainerDied","Data":"0b2ab57c9963092e388609035a78c9a5f76acd9fc7a2b172a5d20e7777c83682"} Jul 08 02:47:30.801296 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:30.801263 2576 generic.go:356] "Generic (PLEG): container finished" podID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerID="f7206cc5382f174d9164d589d01ecb1f241d33d3b5b51318e4b88180e3905cce" exitCode=0 Jul 08 02:47:30.801719 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:30.801336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" event={"ID":"28c098dc-9c05-4116-aaeb-2e5377a4b128","Type":"ContainerDied","Data":"f7206cc5382f174d9164d589d01ecb1f241d33d3b5b51318e4b88180e3905cce"} Jul 08 02:47:31.928329 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:31.928297 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:32.011844 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.011815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-util\") pod \"28c098dc-9c05-4116-aaeb-2e5377a4b128\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " Jul 08 02:47:32.012002 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.011865 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4rdx\" (UniqueName: \"kubernetes.io/projected/28c098dc-9c05-4116-aaeb-2e5377a4b128-kube-api-access-m4rdx\") pod \"28c098dc-9c05-4116-aaeb-2e5377a4b128\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " Jul 08 02:47:32.012002 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.011906 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-bundle\") pod \"28c098dc-9c05-4116-aaeb-2e5377a4b128\" (UID: \"28c098dc-9c05-4116-aaeb-2e5377a4b128\") " Jul 08 02:47:32.012572 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.012535 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-bundle" (OuterVolumeSpecName: "bundle") pod "28c098dc-9c05-4116-aaeb-2e5377a4b128" (UID: "28c098dc-9c05-4116-aaeb-2e5377a4b128"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:47:32.014200 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.014174 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c098dc-9c05-4116-aaeb-2e5377a4b128-kube-api-access-m4rdx" (OuterVolumeSpecName: "kube-api-access-m4rdx") pod "28c098dc-9c05-4116-aaeb-2e5377a4b128" (UID: "28c098dc-9c05-4116-aaeb-2e5377a4b128"). InnerVolumeSpecName "kube-api-access-m4rdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:47:32.016684 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.016659 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-util" (OuterVolumeSpecName: "util") pod "28c098dc-9c05-4116-aaeb-2e5377a4b128" (UID: "28c098dc-9c05-4116-aaeb-2e5377a4b128"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:47:32.112526 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.112447 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-util\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:47:32.112526 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.112473 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rdx\" (UniqueName: \"kubernetes.io/projected/28c098dc-9c05-4116-aaeb-2e5377a4b128-kube-api-access-m4rdx\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:47:32.112526 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.112483 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28c098dc-9c05-4116-aaeb-2e5377a4b128-bundle\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:47:32.811574 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.811545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" event={"ID":"28c098dc-9c05-4116-aaeb-2e5377a4b128","Type":"ContainerDied","Data":"c28000eeaf704dff723a9a6a6ee2cb847dc966e180d35a7ce6fd9911aa466bad"} Jul 08 02:47:32.811759 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.811588 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c28000eeaf704dff723a9a6a6ee2cb847dc966e180d35a7ce6fd9911aa466bad" Jul 08 02:47:32.811759 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:32.811596 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754052cvl" Jul 08 02:47:38.619165 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:38.619133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:40.452923 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:40.452865 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:42.444841 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:42.444793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:42.542627 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:42.542594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:45.213072 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:45.213041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:45.310342 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:45.310311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:45.680848 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:45.680809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:45.882382 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:45.882323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:47.613246 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:47.613210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:48.029531 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.029484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:48.370291 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370220 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v"] Jul 08 02:47:48.370523 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370511 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="util" Jul 08 02:47:48.370567 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370525 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="util" Jul 08 02:47:48.370567 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370537 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="pull" Jul 08 02:47:48.370567 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370543 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="pull" Jul 08 02:47:48.370567 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370549 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="extract" Jul 08 02:47:48.370567 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370554 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="extract" Jul 08 02:47:48.370758 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.370607 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c098dc-9c05-4116-aaeb-2e5377a4b128" containerName="extract" Jul 08 02:47:48.373803 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.373785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.376292 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.376267 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 02:47:48.376465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.376267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wrl79\"" Jul 08 02:47:48.377251 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.377237 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 02:47:48.383760 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.383742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v"] Jul 08 02:47:48.540680 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.540634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br7k\" (UniqueName: \"kubernetes.io/projected/ac6d850f-c9f5-47ff-a31f-cb88bce9fc37-kube-api-access-6br7k\") pod \"cert-manager-cainjector-559cdd8c87-kkm7v\" (UID: \"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.540856 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.540698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac6d850f-c9f5-47ff-a31f-cb88bce9fc37-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kkm7v\" (UID: \"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.641113 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.641084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac6d850f-c9f5-47ff-a31f-cb88bce9fc37-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kkm7v\" (UID: \"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.641520 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.641165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br7k\" (UniqueName: \"kubernetes.io/projected/ac6d850f-c9f5-47ff-a31f-cb88bce9fc37-kube-api-access-6br7k\") pod \"cert-manager-cainjector-559cdd8c87-kkm7v\" (UID: \"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.655776 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.655748 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac6d850f-c9f5-47ff-a31f-cb88bce9fc37-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kkm7v\" (UID: \"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.655898 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.655869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br7k\" (UniqueName: \"kubernetes.io/projected/ac6d850f-c9f5-47ff-a31f-cb88bce9fc37-kube-api-access-6br7k\") pod \"cert-manager-cainjector-559cdd8c87-kkm7v\" (UID: \"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.683694 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.683668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" Jul 08 02:47:48.815154 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.815128 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v"] Jul 08 02:47:48.817441 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:47:48.817416 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac6d850f_c9f5_47ff_a31f_cb88bce9fc37.slice/crio-2472f644d95170054d0a10af03845c411da16076edd7566f1f3e2fe17fd45ba5 WatchSource:0}: Error finding container 2472f644d95170054d0a10af03845c411da16076edd7566f1f3e2fe17fd45ba5: Status 404 returned error can't find the container with id 2472f644d95170054d0a10af03845c411da16076edd7566f1f3e2fe17fd45ba5 Jul 08 02:47:48.877915 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:48.877884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" event={"ID":"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37","Type":"ContainerStarted","Data":"2472f644d95170054d0a10af03845c411da16076edd7566f1f3e2fe17fd45ba5"} Jul 08 02:47:49.066430 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:49.066342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:49.144703 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:49.144668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:52.897300 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:52.897253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" event={"ID":"ac6d850f-c9f5-47ff-a31f-cb88bce9fc37","Type":"ContainerStarted","Data":"4425d96b1961a42df2bdffcee1fd279503574838b7c7aff35250646881ccda60"} Jul 08 02:47:52.913988 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:52.913940 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kkm7v" podStartSLOduration=1.118639719 podStartE2EDuration="4.91392499s" totalImagesPullingTime="3.795285271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:47:52.912154756 +0000 UTC m=+463.190531960" watchObservedRunningTime="2026-07-08 02:47:52.91392499 +0000 UTC m=+463.192302191" Jul 08 02:47:53.440055 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:53.440028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:53.533368 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:53.533331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:54.817923 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:54.817894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:47:56.607349 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.607317 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z4wvc"] Jul 08 02:47:56.609729 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.609714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.612042 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.612020 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rnbxq\"" Jul 08 02:47:56.620244 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.620221 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z4wvc"] Jul 08 02:47:56.710289 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.710261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5j7\" (UniqueName: \"kubernetes.io/projected/4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3-kube-api-access-vz5j7\") pod \"cert-manager-55b4d954d5-z4wvc\" (UID: \"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3\") " pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.710289 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.710290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3-bound-sa-token\") pod \"cert-manager-55b4d954d5-z4wvc\" (UID: \"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3\") " pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.810633 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.810603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5j7\" (UniqueName: \"kubernetes.io/projected/4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3-kube-api-access-vz5j7\") pod \"cert-manager-55b4d954d5-z4wvc\" (UID: \"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3\") " pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.810814 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.810636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3-bound-sa-token\") pod \"cert-manager-55b4d954d5-z4wvc\" (UID: \"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3\") " pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.818632 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.818599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3-bound-sa-token\") pod \"cert-manager-55b4d954d5-z4wvc\" (UID: \"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3\") " pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.818727 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.818630 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5j7\" (UniqueName: \"kubernetes.io/projected/4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3-kube-api-access-vz5j7\") pod \"cert-manager-55b4d954d5-z4wvc\" (UID: \"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3\") " pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:56.918955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:56.918929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z4wvc" Jul 08 02:47:57.046610 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:47:57.046584 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c7d37ae_6d3f_47b6_b476_c6e4de48f9c3.slice/crio-784bc7db62f5f0f901029f89d524314d0ac3cb3521715f2005af589945641f75 WatchSource:0}: Error finding container 784bc7db62f5f0f901029f89d524314d0ac3cb3521715f2005af589945641f75: Status 404 returned error can't find the container with id 784bc7db62f5f0f901029f89d524314d0ac3cb3521715f2005af589945641f75 Jul 08 02:47:57.049803 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:57.049779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z4wvc"] Jul 08 02:47:57.917230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:57.917196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z4wvc" event={"ID":"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3","Type":"ContainerStarted","Data":"94d1bae00081413b77a71be8fb9452537c8b5969a61a8bcddfd716decb93a099"} Jul 08 02:47:57.917230 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:57.917231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z4wvc" event={"ID":"4c7d37ae-6d3f-47b6-b476-c6e4de48f9c3","Type":"ContainerStarted","Data":"784bc7db62f5f0f901029f89d524314d0ac3cb3521715f2005af589945641f75"} Jul 08 02:47:57.933161 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:47:57.933120 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-z4wvc" podStartSLOduration=1.933106128 podStartE2EDuration="1.933106128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:47:57.931830961 +0000 UTC m=+468.210208173" watchObservedRunningTime="2026-07-08 02:47:57.933106128 +0000 UTC m=+468.211483330" Jul 08 02:48:10.307227 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:10.307192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:10.408032 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:10.408000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:17.340789 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:17.340758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:17.443387 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:17.443345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:19.534808 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:19.534778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:24.068971 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:24.068937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:31.814833 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.814799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:31.912239 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.912211 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh"] Jul 08 02:48:31.914483 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.914468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:31.919180 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.919158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:48:31.919344 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.919165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 02:48:31.919447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.919170 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 02:48:31.919447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.919186 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7497w\"" Jul 08 02:48:31.919447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.919215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 02:48:31.919590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.919239 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 02:48:31.941525 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.941494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh"] Jul 08 02:48:31.958015 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.957968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:31.979308 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.979282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b2442b-b69c-402b-9219-57683cc44445-cert\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:31.979449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.979324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7b2442b-b69c-402b-9219-57683cc44445-metrics-cert\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:31.979449 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.979381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7b2442b-b69c-402b-9219-57683cc44445-manager-config\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:31.979546 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:31.979498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j968\" (UniqueName: \"kubernetes.io/projected/d7b2442b-b69c-402b-9219-57683cc44445-kube-api-access-7j968\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.080399 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.080292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7b2442b-b69c-402b-9219-57683cc44445-metrics-cert\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.080399 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.080338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7b2442b-b69c-402b-9219-57683cc44445-manager-config\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.080584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.080432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j968\" (UniqueName: \"kubernetes.io/projected/d7b2442b-b69c-402b-9219-57683cc44445-kube-api-access-7j968\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.080584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.080477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b2442b-b69c-402b-9219-57683cc44445-cert\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.081099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.081079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7b2442b-b69c-402b-9219-57683cc44445-manager-config\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.082916 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.082896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7b2442b-b69c-402b-9219-57683cc44445-metrics-cert\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.083002 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.082968 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b2442b-b69c-402b-9219-57683cc44445-cert\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.092815 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.092796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j968\" (UniqueName: \"kubernetes.io/projected/d7b2442b-b69c-402b-9219-57683cc44445-kube-api-access-7j968\") pod \"lws-controller-manager-56954ff8c6-rz6wh\" (UID: \"d7b2442b-b69c-402b-9219-57683cc44445\") " pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.223680 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.223654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:32.375319 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:32.375295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh"] Jul 08 02:48:32.377081 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:48:32.377055 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b2442b_b69c_402b_9219_57683cc44445.slice/crio-7da1858e3eaed3aea3d369f45b77e2641f718268842d22d3ae72b427c12c0c5d WatchSource:0}: Error finding container 7da1858e3eaed3aea3d369f45b77e2641f718268842d22d3ae72b427c12c0c5d: Status 404 returned error can't find the container with id 7da1858e3eaed3aea3d369f45b77e2641f718268842d22d3ae72b427c12c0c5d Jul 08 02:48:33.062045 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:33.062004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" event={"ID":"d7b2442b-b69c-402b-9219-57683cc44445","Type":"ContainerStarted","Data":"7da1858e3eaed3aea3d369f45b77e2641f718268842d22d3ae72b427c12c0c5d"} Jul 08 02:48:34.250401 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:34.250368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:34.352817 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:34.352783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:34.875963 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:34.875912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.071248 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.071165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" event={"ID":"d7b2442b-b69c-402b-9219-57683cc44445","Type":"ContainerStarted","Data":"ddece5e50311d00eafce02f50e4aad77c37d7cbe5a15d43f9b4edac8fc9329a0"} Jul 08 02:48:35.071423 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.071281 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:35.089241 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.089190 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" podStartSLOduration=1.673125678 podStartE2EDuration="4.089177236s" totalImagesPullingTime="2.416051558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:48:35.088515356 +0000 UTC m=+505.366892558" watchObservedRunningTime="2026-07-08 02:48:35.089177236 +0000 UTC m=+505.367554438" Jul 08 02:48:35.094298 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.094272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.199307 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.199279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.253858 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.253830 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml"] Jul 08 02:48:35.256369 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.256340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.269933 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.269914 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 02:48:35.270870 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.270855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 02:48:35.270927 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.270893 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7n7w2\"" Jul 08 02:48:35.270970 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.270942 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 02:48:35.270970 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.270957 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 02:48:35.279770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.279742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml"] Jul 08 02:48:35.313350 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.313319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50770948-121c-40b6-a313-7d461aebc60b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.313520 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.313386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50770948-121c-40b6-a313-7d461aebc60b-tmp\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.313520 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.313412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhgx\" (UniqueName: \"kubernetes.io/projected/50770948-121c-40b6-a313-7d461aebc60b-kube-api-access-2hhgx\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.313520 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.313440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50770948-121c-40b6-a313-7d461aebc60b-manifests\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.313642 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.313552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50770948-121c-40b6-a313-7d461aebc60b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.409803 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.409767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.413958 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.413937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50770948-121c-40b6-a313-7d461aebc60b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.414054 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.413976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50770948-121c-40b6-a313-7d461aebc60b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.414113 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.414097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50770948-121c-40b6-a313-7d461aebc60b-tmp\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.414163 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.414123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhgx\" (UniqueName: \"kubernetes.io/projected/50770948-121c-40b6-a313-7d461aebc60b-kube-api-access-2hhgx\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.414163 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.414148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50770948-121c-40b6-a313-7d461aebc60b-manifests\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.414582 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.414560 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50770948-121c-40b6-a313-7d461aebc60b-manifests\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.416430 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.416404 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50770948-121c-40b6-a313-7d461aebc60b-tmp\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.416684 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.416667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50770948-121c-40b6-a313-7d461aebc60b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.416684 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.416676 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50770948-121c-40b6-a313-7d461aebc60b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.422000 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.421982 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhgx\" (UniqueName: \"kubernetes.io/projected/50770948-121c-40b6-a313-7d461aebc60b-kube-api-access-2hhgx\") pod \"opendatahub-operator-controller-manager-6fbb74d9dd-8ncml\" (UID: \"50770948-121c-40b6-a313-7d461aebc60b\") " pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.514864 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.514828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.566010 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.565975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:35.619828 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.619792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.710392 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.710305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.810099 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.810067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:35.908284 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:35.908259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml"] Jul 08 02:48:35.911692 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:48:35.911658 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50770948_121c_40b6_a313_7d461aebc60b.slice/crio-c78725ec163fe3c387e9ef2666acbbd7edbe36df4518dbabaa1b08dafc63da70 WatchSource:0}: Error finding container c78725ec163fe3c387e9ef2666acbbd7edbe36df4518dbabaa1b08dafc63da70: Status 404 returned error can't find the container with id c78725ec163fe3c387e9ef2666acbbd7edbe36df4518dbabaa1b08dafc63da70 Jul 08 02:48:36.076801 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:36.076731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" event={"ID":"50770948-121c-40b6-a313-7d461aebc60b","Type":"ContainerStarted","Data":"c78725ec163fe3c387e9ef2666acbbd7edbe36df4518dbabaa1b08dafc63da70"} Jul 08 02:48:39.094028 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:39.093995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" event={"ID":"50770948-121c-40b6-a313-7d461aebc60b","Type":"ContainerStarted","Data":"47ce6ce6c833ad8c7d2031e8cd277bef87ccbc4e886e3abffc4ae578332a5760"} Jul 08 02:48:40.100220 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:40.100186 2576 generic.go:356] "Generic (PLEG): container finished" podID="50770948-121c-40b6-a313-7d461aebc60b" containerID="47ce6ce6c833ad8c7d2031e8cd277bef87ccbc4e886e3abffc4ae578332a5760" exitCode=0 Jul 08 02:48:40.100642 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:40.100241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" event={"ID":"50770948-121c-40b6-a313-7d461aebc60b","Type":"ContainerDied","Data":"47ce6ce6c833ad8c7d2031e8cd277bef87ccbc4e886e3abffc4ae578332a5760"} Jul 08 02:48:41.107015 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:41.106976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" event={"ID":"50770948-121c-40b6-a313-7d461aebc60b","Type":"ContainerStarted","Data":"592b29b93710c8733584b3d6436960e33db8649b89a575d99aa4ae87499997f5"} Jul 08 02:48:41.107519 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:41.107096 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:41.129472 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:41.129413 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" podStartSLOduration=1.710408094 podStartE2EDuration="6.129392327s" totalImagesPullingTime="3.415984068s" totalInitContainerRuntime="1.003000165s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:48:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:48:41.126510755 +0000 UTC m=+511.404887958" watchObservedRunningTime="2026-07-08 02:48:41.129392327 +0000 UTC m=+511.407769531" Jul 08 02:48:45.796939 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:45.796900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:46.079281 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:46.079216 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56954ff8c6-rz6wh" Jul 08 02:48:52.113588 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:52.113560 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fbb74d9dd-8ncml" Jul 08 02:48:54.185200 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:54.185163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:54.308068 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:54.308035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:54.742666 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:54.742631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:54.988248 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:54.988213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:55.089177 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:55.089100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:55.188059 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:55.188023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:55.282397 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:55.282340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:55.776570 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:55.776538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:55.958344 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:55.958298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:56.110488 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:56.110406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:56.422174 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:56.422141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:56.502508 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:56.502436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:48:57.594102 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:48:57.594069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:01.695616 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:01.695528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:01.795689 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:01.795655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:01.987487 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:01.987403 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n"] Jul 08 02:49:01.990635 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:01.990613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:01.993720 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:01.993703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bmnt8\"" Jul 08 02:49:01.994291 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:01.994279 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 02:49:02.013851 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.013830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n"] Jul 08 02:49:02.021020 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.021000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:02.041571 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2m7h\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-kube-api-access-g2m7h\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041676 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041676 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041676 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801fc730-5e67-43f0-8b81-433d3921dcc8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041805 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041805 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041805 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041923 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.041923 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.041870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142334 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2m7h\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-kube-api-access-g2m7h\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801fc730-5e67-43f0-8b81-433d3921dcc8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142890 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.142890 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142763 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.143042 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.143042 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.142936 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.143189 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.143163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.143294 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.143196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801fc730-5e67-43f0-8b81-433d3921dcc8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.144883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.144864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.145474 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.145455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.171054 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.171029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.172156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.172139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2m7h\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-kube-api-access-g2m7h\") pod \"data-science-gateway-data-science-gateway-class-548557588-q687n\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.192490 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.192466 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk"] Jul 08 02:49:02.197184 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.197168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.231570 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.231543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk"] Jul 08 02:49:02.243402 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243402 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bd1918-d796-4039-a592-ccefd26d35c7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bd1918-d796-4039-a592-ccefd26d35c7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243584 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnqf\" (UniqueName: \"kubernetes.io/projected/90bd1918-d796-4039-a592-ccefd26d35c7-kube-api-access-hjnqf\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243780 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.243780 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.243636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bd1918-d796-4039-a592-ccefd26d35c7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.300899 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.300871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:02.344878 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.344847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345047 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.344903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bd1918-d796-4039-a592-ccefd26d35c7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345047 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.344928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnqf\" (UniqueName: \"kubernetes.io/projected/90bd1918-d796-4039-a592-ccefd26d35c7-kube-api-access-hjnqf\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345047 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345214 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bd1918-d796-4039-a592-ccefd26d35c7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345214 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345317 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.345508 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bd1918-d796-4039-a592-ccefd26d35c7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.346180 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.345923 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.346180 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.346041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bd1918-d796-4039-a592-ccefd26d35c7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.346180 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.346129 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.346460 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.346226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.348827 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.348712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bd1918-d796-4039-a592-ccefd26d35c7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.348946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.348919 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bd1918-d796-4039-a592-ccefd26d35c7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.362756 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.362703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnqf\" (UniqueName: \"kubernetes.io/projected/90bd1918-d796-4039-a592-ccefd26d35c7-kube-api-access-hjnqf\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.362884 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.362759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bd1918-d796-4039-a592-ccefd26d35c7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jtvbk\" (UID: \"90bd1918-d796-4039-a592-ccefd26d35c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.508932 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.508862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:02.512888 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:49:02.512863 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801fc730_5e67_43f0_8b81_433d3921dcc8.slice/crio-bcd5d7c1f10a9336636e9946486438cad55d95149153a9d89d4e9792edcb5af1 WatchSource:0}: Error finding container bcd5d7c1f10a9336636e9946486438cad55d95149153a9d89d4e9792edcb5af1: Status 404 returned error can't find the container with id bcd5d7c1f10a9336636e9946486438cad55d95149153a9d89d4e9792edcb5af1 Jul 08 02:49:02.519549 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.514298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n"] Jul 08 02:49:02.540545 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.540500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:02.661272 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.661244 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk"] Jul 08 02:49:02.663894 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:49:02.663869 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bd1918_d796_4039_a592_ccefd26d35c7.slice/crio-a312d3f4c1d299acfe3f10ae59177ece0dd51181709767ff5d886d5833d647b6 WatchSource:0}: Error finding container a312d3f4c1d299acfe3f10ae59177ece0dd51181709767ff5d886d5833d647b6: Status 404 returned error can't find the container with id a312d3f4c1d299acfe3f10ae59177ece0dd51181709767ff5d886d5833d647b6 Jul 08 02:49:02.706665 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.706639 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk"] Jul 08 02:49:02.711097 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.711079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.714035 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.714010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-45thl\"" Jul 08 02:49:02.714144 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.714040 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 02:49:02.714296 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.714282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 02:49:02.715855 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.715838 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 02:49:02.726610 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.726591 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk"] Jul 08 02:49:02.748011 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.747985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.748104 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.748034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.748142 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.748108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.748177 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.748163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.748239 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.748225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.748284 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.748265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.748336 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.748288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrgn\" (UniqueName: \"kubernetes.io/projected/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-kube-api-access-kdrgn\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849618 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849618 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrgn\" (UniqueName: \"kubernetes.io/projected/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-kube-api-access-kdrgn\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.849802 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.849728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.850443 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.850345 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.851860 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.851840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.852040 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.852021 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.852444 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.852423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.852528 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.852446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.868759 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.868736 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.872629 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.872610 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrgn\" (UniqueName: \"kubernetes.io/projected/c9c45844-eeeb-4fe4-9454-27cfce5ddfee-kube-api-access-kdrgn\") pod \"istiod-openshift-gateway-94bb8fbfd-jktwk\" (UID: \"c9c45844-eeeb-4fe4-9454-27cfce5ddfee\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:02.985922 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:02.985885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:03.021952 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:03.021928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:03.165483 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:03.165451 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk"] Jul 08 02:49:03.210064 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:03.210027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" event={"ID":"801fc730-5e67-43f0-8b81-433d3921dcc8","Type":"ContainerStarted","Data":"bcd5d7c1f10a9336636e9946486438cad55d95149153a9d89d4e9792edcb5af1"} Jul 08 02:49:03.211639 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:03.211612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" event={"ID":"c9c45844-eeeb-4fe4-9454-27cfce5ddfee","Type":"ContainerStarted","Data":"f63ff366aa97b1488bc5ab8ca3667713661fdee8801ab03dbdcefe83b2e10415"} Jul 08 02:49:03.212956 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:03.212932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" event={"ID":"90bd1918-d796-4039-a592-ccefd26d35c7","Type":"ContainerStarted","Data":"a312d3f4c1d299acfe3f10ae59177ece0dd51181709767ff5d886d5833d647b6"} Jul 08 02:49:05.660065 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:05.660012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:05.760616 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:05.760555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:05.893649 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:05.893587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:06.667825 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.667782 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.667825 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.667807 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.668234 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.667874 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.668234 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.667877 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.668234 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.667903 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.674030 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.674008 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.674115 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.674063 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.674115 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.674090 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 02:49:06.724625 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.724590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:06.943024 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:06.942927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:07.058966 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.058936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:07.163004 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.162969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:07.240335 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.240233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" event={"ID":"c9c45844-eeeb-4fe4-9454-27cfce5ddfee","Type":"ContainerStarted","Data":"addbc77e85d4a51da00b5131949266cb0cffc461c511f7d7ca4a9777166e5a6e"} Jul 08 02:49:07.240566 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.240391 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:07.242492 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.242465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" event={"ID":"90bd1918-d796-4039-a592-ccefd26d35c7","Type":"ContainerStarted","Data":"88a52a8155d7dbfe887283b42d2e470d9621c73c9ff5bc1e59ac4c3d2bec346c"} Jul 08 02:49:07.242855 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.242821 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-jktwk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 02:49:07.242974 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.242876 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" podUID="c9c45844-eeeb-4fe4-9454-27cfce5ddfee" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 02:49:07.244556 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.244529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" event={"ID":"801fc730-5e67-43f0-8b81-433d3921dcc8","Type":"ContainerStarted","Data":"709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083"} Jul 08 02:49:07.263382 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.263262 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" podStartSLOduration=1.769172947 podStartE2EDuration="5.263242352s" totalImagesPullingTime="3.494069405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:49:07.262349069 +0000 UTC m=+537.540726300" watchObservedRunningTime="2026-07-08 02:49:07.263242352 +0000 UTC m=+537.541619555" Jul 08 02:49:07.288747 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.288667 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" podStartSLOduration=1.2874571129999999 podStartE2EDuration="5.288647381s" totalImagesPullingTime="4.001190268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:49:07.286945984 +0000 UTC m=+537.565323187" watchObservedRunningTime="2026-07-08 02:49:07.288647381 +0000 UTC m=+537.567024579" Jul 08 02:49:07.301644 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.301614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:07.302816 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.302783 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-q687n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 08 02:49:07.302946 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.302837 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 08 02:49:07.319569 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.319505 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" podStartSLOduration=2.16683615 podStartE2EDuration="6.319482328s" totalImagesPullingTime="4.152646178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:49:07.31290123 +0000 UTC m=+537.591278431" watchObservedRunningTime="2026-07-08 02:49:07.319482328 +0000 UTC m=+537.597859532" Jul 08 02:49:07.509070 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.508995 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:07.513678 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:07.513653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:08.249001 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:08.248972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:08.249517 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:08.249496 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jktwk" Jul 08 02:49:08.250030 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:08.250012 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jtvbk" Jul 08 02:49:08.301722 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:08.301688 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-q687n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 08 02:49:08.301966 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:08.301929 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 08 02:49:08.329217 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:08.329183 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n"] Jul 08 02:49:09.301534 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:09.301505 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-q687n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 08 02:49:09.301888 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:09.301555 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 08 02:49:10.258149 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:10.258107 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" containerID="cri-o://709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083" gracePeriod=30 Jul 08 02:49:15.024991 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.024955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:15.511398 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.511375 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:15.570163 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570098 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-socket\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570163 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-podinfo\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570163 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2m7h\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-kube-api-access-g2m7h\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570398 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570188 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-envoy\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570398 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-data\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570514 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570423 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801fc730-5e67-43f0-8b81-433d3921dcc8-istiod-ca-cert\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570514 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570450 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:49:15.570514 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570464 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-token\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570514 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570491 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-credential-socket\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570723 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570518 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-certs\") pod \"801fc730-5e67-43f0-8b81-433d3921dcc8\" (UID: \"801fc730-5e67-43f0-8b81-433d3921dcc8\") " Jul 08 02:49:15.570723 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570541 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-data" (OuterVolumeSpecName: "istio-data") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:49:15.570817 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570721 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:49:15.570817 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570757 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:49:15.570817 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570792 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801fc730-5e67-43f0-8b81-433d3921dcc8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:49:15.570951 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570862 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.570951 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570882 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.570951 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570896 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801fc730-5e67-43f0-8b81-433d3921dcc8-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.570951 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570909 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.570951 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.570923 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.572704 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.572679 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 02:49:15.572851 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.572800 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-kube-api-access-g2m7h" (OuterVolumeSpecName: "kube-api-access-g2m7h") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "kube-api-access-g2m7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:49:15.573100 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.573077 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:49:15.573100 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.573090 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "801fc730-5e67-43f0-8b81-433d3921dcc8" (UID: "801fc730-5e67-43f0-8b81-433d3921dcc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:49:15.671647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.671615 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.671647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.671641 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.671647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.671651 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2m7h\" (UniqueName: \"kubernetes.io/projected/801fc730-5e67-43f0-8b81-433d3921dcc8-kube-api-access-g2m7h\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:15.671860 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:15.671660 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801fc730-5e67-43f0-8b81-433d3921dcc8-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:49:16.287622 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.287596 2576 generic.go:356] "Generic (PLEG): container finished" podID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerID="709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083" exitCode=0 Jul 08 02:49:16.287905 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.287636 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" Jul 08 02:49:16.287905 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.287636 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" event={"ID":"801fc730-5e67-43f0-8b81-433d3921dcc8","Type":"ContainerDied","Data":"709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083"} Jul 08 02:49:16.287905 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.287778 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n" event={"ID":"801fc730-5e67-43f0-8b81-433d3921dcc8","Type":"ContainerDied","Data":"bcd5d7c1f10a9336636e9946486438cad55d95149153a9d89d4e9792edcb5af1"} Jul 08 02:49:16.287905 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.287814 2576 scope.go:117] "RemoveContainer" containerID="709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083" Jul 08 02:49:16.296721 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.296706 2576 scope.go:117] "RemoveContainer" containerID="709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083" Jul 08 02:49:16.296993 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:49:16.296974 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083\": container with ID starting with 709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083 not found: ID does not exist" containerID="709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083" Jul 08 02:49:16.297040 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.297001 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083"} err="failed to get container status \"709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083\": rpc error: code = NotFound desc = could not find container \"709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083\": container with ID starting with 709a5d0c3e5069ead24255877f977b114a72ac6f3b8500adf77a648a08285083 not found: ID does not exist" Jul 08 02:49:16.304833 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.304813 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n"] Jul 08 02:49:16.310505 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:16.310482 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-q687n"] Jul 08 02:49:18.278577 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:18.278543 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" path="/var/lib/kubelet/pods/801fc730-5e67-43f0-8b81-433d3921dcc8/volumes" Jul 08 02:49:55.660974 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:55.660911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:55.782487 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:55.782453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:49:55.884265 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:49:55.884235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:06.773952 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:06.773915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:08.618245 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:08.618211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:08.678214 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:08.678184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:10.138275 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:10.138235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:50:10.141425 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:10.141399 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:50:17.523315 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:17.523272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:17.578490 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:17.578465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:24.624885 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:24.624849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:25.232899 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:25.232872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:30.053448 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:30.053414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:32.173587 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:32.173534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:32.636097 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:32.636071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:34.492219 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:34.492189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:36.271269 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:36.271243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:37.846896 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:37.846867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:39.066708 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:39.066673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:41.033398 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:41.033370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:41.135760 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:41.135728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:41.434293 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:41.434259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:41.535037 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:41.535008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:44.843507 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:44.843482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:44.943172 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:44.943145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:47.050430 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:47.050399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:47.209545 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:47.209517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:48.230264 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:48.230233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:48.283015 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:48.282991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:53.906786 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:53.906756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:54.007661 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:54.007632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:59.617198 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:59.617166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:50:59.721728 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:50:59.721704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:00.224523 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:00.224488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:00.303579 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:00.303551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:03.616557 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:03.616522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:03.718130 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:03.718095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:07.018511 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:07.018482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:07.119740 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:07.119712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:12.805172 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:12.805095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:12.868590 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:12.868561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:15.318484 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:15.318450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:15.398348 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:15.398313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:15.466164 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:15.466135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:16.061593 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:16.061559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:16.162040 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:16.162003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:25.875103 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:25.875072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:25.974909 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:25.974876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:26.194980 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:26.194906 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c66fb6bb6-57vt4"] Jul 08 02:51:26.313652 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:26.313624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:26.408647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:26.408619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:26.461154 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:26.461097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:38.314517 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:38.314484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:38.427717 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:38.427683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:38.530145 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:38.530112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:39.228191 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:39.228163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:39.329880 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:39.329849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:51.217421 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.217328 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c66fb6bb6-57vt4" podUID="d8946432-a817-48c2-a17e-3ebed7649536" containerName="console" containerID="cri-o://e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d" gracePeriod=15 Jul 08 02:51:51.464290 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.464268 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:51:51.601546 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5kjb\" (UniqueName: \"kubernetes.io/projected/d8946432-a817-48c2-a17e-3ebed7649536-kube-api-access-x5kjb\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.601546 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601526 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-console-config\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.601764 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601577 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-oauth-serving-cert\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.601764 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-trusted-ca-bundle\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.601764 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601630 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-serving-cert\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.601764 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-oauth-config\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.601764 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.601671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-service-ca\") pod \"d8946432-a817-48c2-a17e-3ebed7649536\" (UID: \"d8946432-a817-48c2-a17e-3ebed7649536\") " Jul 08 02:51:51.602064 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.602033 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:51:51.602126 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.602098 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:51:51.602184 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.602040 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-console-config" (OuterVolumeSpecName: "console-config") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:51:51.602256 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.602227 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:51:51.603886 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.603860 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8946432-a817-48c2-a17e-3ebed7649536-kube-api-access-x5kjb" (OuterVolumeSpecName: "kube-api-access-x5kjb") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "kube-api-access-x5kjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:51:51.603886 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.603876 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:51:51.604032 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.603905 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8946432-a817-48c2-a17e-3ebed7649536" (UID: "d8946432-a817-48c2-a17e-3ebed7649536"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:51:51.702611 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702575 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-console-config\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.702611 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702607 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-oauth-serving-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.702611 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702622 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-trusted-ca-bundle\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.702847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702634 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-serving-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.702847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702644 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8946432-a817-48c2-a17e-3ebed7649536-console-oauth-config\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.702847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702653 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8946432-a817-48c2-a17e-3ebed7649536-service-ca\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.702847 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.702661 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kjb\" (UniqueName: \"kubernetes.io/projected/d8946432-a817-48c2-a17e-3ebed7649536-kube-api-access-x5kjb\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:51:51.974421 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.974377 2576 generic.go:356] "Generic (PLEG): container finished" podID="d8946432-a817-48c2-a17e-3ebed7649536" containerID="e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d" exitCode=0 Jul 08 02:51:51.974421 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.974425 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c66fb6bb6-57vt4" Jul 08 02:51:51.974679 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.974438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c66fb6bb6-57vt4" event={"ID":"d8946432-a817-48c2-a17e-3ebed7649536","Type":"ContainerDied","Data":"e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d"} Jul 08 02:51:51.974679 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.974507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c66fb6bb6-57vt4" event={"ID":"d8946432-a817-48c2-a17e-3ebed7649536","Type":"ContainerDied","Data":"b0eb3405bb686fe98d03925ed49613bceee906df9311e8b21b023b066867839b"} Jul 08 02:51:51.974679 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.974536 2576 scope.go:117] "RemoveContainer" containerID="e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d" Jul 08 02:51:51.983213 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.983197 2576 scope.go:117] "RemoveContainer" containerID="e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d" Jul 08 02:51:51.983495 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:51:51.983478 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d\": container with ID starting with e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d not found: ID does not exist" containerID="e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d" Jul 08 02:51:51.983560 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.983503 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d"} err="failed to get container status \"e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d\": rpc error: code = NotFound desc = could not find container \"e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d\": container with ID starting with e09ef1fd08860507b26137d6243ae561114a2021f69fa040ad6f330611c01e2d not found: ID does not exist" Jul 08 02:51:51.994728 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.994708 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c66fb6bb6-57vt4"] Jul 08 02:51:51.999783 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:51.999761 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c66fb6bb6-57vt4"] Jul 08 02:51:52.278239 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:52.278168 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8946432-a817-48c2-a17e-3ebed7649536" path="/var/lib/kubelet/pods/d8946432-a817-48c2-a17e-3ebed7649536/volumes" Jul 08 02:51:57.988744 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:57.988695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:58.084882 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.084845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:58.438284 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438255 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd"] Jul 08 02:51:58.438737 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438722 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" Jul 08 02:51:58.438786 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438742 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" Jul 08 02:51:58.438821 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438785 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8946432-a817-48c2-a17e-3ebed7649536" containerName="console" Jul 08 02:51:58.438821 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438794 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8946432-a817-48c2-a17e-3ebed7649536" containerName="console" Jul 08 02:51:58.438881 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438870 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="801fc730-5e67-43f0-8b81-433d3921dcc8" containerName="istio-proxy" Jul 08 02:51:58.438915 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.438885 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8946432-a817-48c2-a17e-3ebed7649536" containerName="console" Jul 08 02:51:58.443635 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.443608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.446193 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.446169 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 02:51:58.446447 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.446172 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 02:51:58.447143 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.447123 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 02:51:58.447346 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.447332 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-j5mcg\"" Jul 08 02:51:58.449379 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.449338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd"] Jul 08 02:51:58.466141 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.466118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:51:58.561834 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.561808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8d6z\" (UniqueName: \"kubernetes.io/projected/7492f611-7482-4854-a97b-ff2a03b9f043-kube-api-access-q8d6z\") pod \"postgres-69fb4594cf-rkbfd\" (UID: \"7492f611-7482-4854-a97b-ff2a03b9f043\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.561958 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.561877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7492f611-7482-4854-a97b-ff2a03b9f043-data\") pod \"postgres-69fb4594cf-rkbfd\" (UID: \"7492f611-7482-4854-a97b-ff2a03b9f043\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.663158 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.663128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8d6z\" (UniqueName: \"kubernetes.io/projected/7492f611-7482-4854-a97b-ff2a03b9f043-kube-api-access-q8d6z\") pod \"postgres-69fb4594cf-rkbfd\" (UID: \"7492f611-7482-4854-a97b-ff2a03b9f043\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.663276 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.663188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7492f611-7482-4854-a97b-ff2a03b9f043-data\") pod \"postgres-69fb4594cf-rkbfd\" (UID: \"7492f611-7482-4854-a97b-ff2a03b9f043\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.663514 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.663499 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7492f611-7482-4854-a97b-ff2a03b9f043-data\") pod \"postgres-69fb4594cf-rkbfd\" (UID: \"7492f611-7482-4854-a97b-ff2a03b9f043\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.671872 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.671850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8d6z\" (UniqueName: \"kubernetes.io/projected/7492f611-7482-4854-a97b-ff2a03b9f043-kube-api-access-q8d6z\") pod \"postgres-69fb4594cf-rkbfd\" (UID: \"7492f611-7482-4854-a97b-ff2a03b9f043\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.755223 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.755158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:51:58.885629 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.885594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd"] Jul 08 02:51:58.888431 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:51:58.888397 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7492f611_7482_4854_a97b_ff2a03b9f043.slice/crio-9ee0a2ca40abb35a73c9bcabcbcc0340486e6c4c3baccdd6dc11c7b4e67c6f78 WatchSource:0}: Error finding container 9ee0a2ca40abb35a73c9bcabcbcc0340486e6c4c3baccdd6dc11c7b4e67c6f78: Status 404 returned error can't find the container with id 9ee0a2ca40abb35a73c9bcabcbcc0340486e6c4c3baccdd6dc11c7b4e67c6f78 Jul 08 02:51:58.889929 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:58.889909 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:51:59.009473 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:51:59.009395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" event={"ID":"7492f611-7482-4854-a97b-ff2a03b9f043","Type":"ContainerStarted","Data":"9ee0a2ca40abb35a73c9bcabcbcc0340486e6c4c3baccdd6dc11c7b4e67c6f78"} Jul 08 02:52:05.041133 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:05.041047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" event={"ID":"7492f611-7482-4854-a97b-ff2a03b9f043","Type":"ContainerStarted","Data":"e0327d01e9a7ad8df7cc805ff75c8b7a04b3da3edb7968462444608c7cbe7a03"} Jul 08 02:52:05.041529 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:05.041192 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:52:05.057115 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:05.057065 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" podStartSLOduration=1.273154419 podStartE2EDuration="7.057049582s" totalImagesPullingTime="5.783895163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:52:05.056538761 +0000 UTC m=+715.334916170" watchObservedRunningTime="2026-07-08 02:52:05.057049582 +0000 UTC m=+715.335426783" Jul 08 02:52:11.075761 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:11.075725 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rkbfd" Jul 08 02:52:11.384533 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:11.384502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:11.627683 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:11.627649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:17.783507 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.783470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:17.796616 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.796585 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f7768d9d9-cpvjk"] Jul 08 02:52:17.819560 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.819526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f7768d9d9-cpvjk"] Jul 08 02:52:17.819711 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.819643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:17.822199 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.822178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 02:52:17.822598 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.822581 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 02:52:17.822696 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.822621 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ws2\"" Jul 08 02:52:17.846622 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.846599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:17.903639 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.903610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:17.934490 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.934461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bfd2fb4-1987-4e98-8805-414b2fb701c0-webhook-cert\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:17.934490 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.934491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxqc\" (UniqueName: \"kubernetes.io/projected/2bfd2fb4-1987-4e98-8805-414b2fb701c0-kube-api-access-6pxqc\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:17.934692 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.934526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd2fb4-1987-4e98-8805-414b2fb701c0-service-ca\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:17.964731 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.964704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-664c944698-mqxlj"] Jul 08 02:52:17.968416 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.968394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:17.978647 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.978622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:17.978968 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:17.978939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664c944698-mqxlj"] Jul 08 02:52:18.035672 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.035600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bfd2fb4-1987-4e98-8805-414b2fb701c0-webhook-cert\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.035672 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.035633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxqc\" (UniqueName: \"kubernetes.io/projected/2bfd2fb4-1987-4e98-8805-414b2fb701c0-kube-api-access-6pxqc\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.035672 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.035666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd2fb4-1987-4e98-8805-414b2fb701c0-service-ca\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.035932 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.035690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c830238e-9adb-4338-8377-08073ff115d3-service-ca\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.035932 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.035727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c830238e-9adb-4338-8377-08073ff115d3-webhook-cert\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.035932 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.035750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vpm\" (UniqueName: \"kubernetes.io/projected/c830238e-9adb-4338-8377-08073ff115d3-kube-api-access-b2vpm\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.036414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.036393 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd2fb4-1987-4e98-8805-414b2fb701c0-service-ca\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.038151 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.038128 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bfd2fb4-1987-4e98-8805-414b2fb701c0-webhook-cert\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.044267 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.044247 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxqc\" (UniqueName: \"kubernetes.io/projected/2bfd2fb4-1987-4e98-8805-414b2fb701c0-kube-api-access-6pxqc\") pod \"maas-controller-6f7768d9d9-cpvjk\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.130117 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.130081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:18.136696 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.136668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c830238e-9adb-4338-8377-08073ff115d3-service-ca\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.136781 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.136755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c830238e-9adb-4338-8377-08073ff115d3-webhook-cert\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.136830 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.136792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vpm\" (UniqueName: \"kubernetes.io/projected/c830238e-9adb-4338-8377-08073ff115d3-kube-api-access-b2vpm\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.137311 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.137288 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c830238e-9adb-4338-8377-08073ff115d3-service-ca\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.139803 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.139780 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c830238e-9adb-4338-8377-08073ff115d3-webhook-cert\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.145918 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.145886 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vpm\" (UniqueName: \"kubernetes.io/projected/c830238e-9adb-4338-8377-08073ff115d3-kube-api-access-b2vpm\") pod \"maas-controller-664c944698-mqxlj\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.263868 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.263822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f7768d9d9-cpvjk"] Jul 08 02:52:18.266348 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:52:18.266314 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfd2fb4_1987_4e98_8805_414b2fb701c0.slice/crio-77de1a02122ea38e3ae0620707dd3106008a36935d26b86fa07140e8a4633502 WatchSource:0}: Error finding container 77de1a02122ea38e3ae0620707dd3106008a36935d26b86fa07140e8a4633502: Status 404 returned error can't find the container with id 77de1a02122ea38e3ae0620707dd3106008a36935d26b86fa07140e8a4633502 Jul 08 02:52:18.278614 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.278590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:18.409296 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:18.409275 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664c944698-mqxlj"] Jul 08 02:52:18.411516 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:52:18.411492 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc830238e_9adb_4338_8377_08073ff115d3.slice/crio-ee9ef43527025e2907163bbf86d9dca01c9c2c9b4350ae66fe03a7162918e99f WatchSource:0}: Error finding container ee9ef43527025e2907163bbf86d9dca01c9c2c9b4350ae66fe03a7162918e99f: Status 404 returned error can't find the container with id ee9ef43527025e2907163bbf86d9dca01c9c2c9b4350ae66fe03a7162918e99f Jul 08 02:52:19.105321 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:19.105277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664c944698-mqxlj" event={"ID":"c830238e-9adb-4338-8377-08073ff115d3","Type":"ContainerStarted","Data":"ee9ef43527025e2907163bbf86d9dca01c9c2c9b4350ae66fe03a7162918e99f"} Jul 08 02:52:19.106786 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:19.106717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" event={"ID":"2bfd2fb4-1987-4e98-8805-414b2fb701c0","Type":"ContainerStarted","Data":"77de1a02122ea38e3ae0620707dd3106008a36935d26b86fa07140e8a4633502"} Jul 08 02:52:20.735963 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:20.735882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:20.841877 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:20.841843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:21.121504 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:21.121402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664c944698-mqxlj" event={"ID":"c830238e-9adb-4338-8377-08073ff115d3","Type":"ContainerStarted","Data":"9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b"} Jul 08 02:52:21.121504 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:21.121467 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:21.123120 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:21.123091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" event={"ID":"2bfd2fb4-1987-4e98-8805-414b2fb701c0","Type":"ContainerStarted","Data":"c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba"} Jul 08 02:52:21.123264 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:21.123151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:21.140184 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:21.140132 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-664c944698-mqxlj" podStartSLOduration=2.063748188 podStartE2EDuration="4.140119698s" totalImagesPullingTime="2.07637151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:52:21.138989114 +0000 UTC m=+731.417366316" watchObservedRunningTime="2026-07-08 02:52:21.140119698 +0000 UTC m=+731.418496877" Jul 08 02:52:21.156209 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:21.156165 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" podStartSLOduration=1.9444799179999999 podStartE2EDuration="4.156149938s" totalImagesPullingTime="2.21167002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:52:21.155503573 +0000 UTC m=+731.433880801" watchObservedRunningTime="2026-07-08 02:52:21.156149938 +0000 UTC m=+731.434527139" Jul 08 02:52:22.928787 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:22.928753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:27.196238 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:27.196206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:27.294826 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:27.294798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:32.134095 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.134066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:32.134569 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.134223 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:52:32.194799 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.194769 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f7768d9d9-cpvjk"] Jul 08 02:52:32.195042 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.195010 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" podUID="2bfd2fb4-1987-4e98-8805-414b2fb701c0" containerName="manager" containerID="cri-o://c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba" gracePeriod=30 Jul 08 02:52:32.441311 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.441287 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:32.571861 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.571835 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd2fb4-1987-4e98-8805-414b2fb701c0-service-ca\") pod \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " Jul 08 02:52:32.572043 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.571882 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bfd2fb4-1987-4e98-8805-414b2fb701c0-webhook-cert\") pod \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " Jul 08 02:52:32.572043 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.571944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pxqc\" (UniqueName: \"kubernetes.io/projected/2bfd2fb4-1987-4e98-8805-414b2fb701c0-kube-api-access-6pxqc\") pod \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\" (UID: \"2bfd2fb4-1987-4e98-8805-414b2fb701c0\") " Jul 08 02:52:32.572252 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.572227 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfd2fb4-1987-4e98-8805-414b2fb701c0-service-ca" (OuterVolumeSpecName: "service-ca") pod "2bfd2fb4-1987-4e98-8805-414b2fb701c0" (UID: "2bfd2fb4-1987-4e98-8805-414b2fb701c0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:52:32.574184 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.574159 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfd2fb4-1987-4e98-8805-414b2fb701c0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2bfd2fb4-1987-4e98-8805-414b2fb701c0" (UID: "2bfd2fb4-1987-4e98-8805-414b2fb701c0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:52:32.574281 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.574247 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfd2fb4-1987-4e98-8805-414b2fb701c0-kube-api-access-6pxqc" (OuterVolumeSpecName: "kube-api-access-6pxqc") pod "2bfd2fb4-1987-4e98-8805-414b2fb701c0" (UID: "2bfd2fb4-1987-4e98-8805-414b2fb701c0"). InnerVolumeSpecName "kube-api-access-6pxqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:52:32.673466 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.673389 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pxqc\" (UniqueName: \"kubernetes.io/projected/2bfd2fb4-1987-4e98-8805-414b2fb701c0-kube-api-access-6pxqc\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:52:32.673466 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.673416 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd2fb4-1987-4e98-8805-414b2fb701c0-service-ca\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:52:32.673466 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:32.673426 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bfd2fb4-1987-4e98-8805-414b2fb701c0-webhook-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:52:33.178778 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.178735 2576 generic.go:356] "Generic (PLEG): container finished" podID="2bfd2fb4-1987-4e98-8805-414b2fb701c0" containerID="c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba" exitCode=0 Jul 08 02:52:33.179346 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.178797 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" Jul 08 02:52:33.179346 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.178821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" event={"ID":"2bfd2fb4-1987-4e98-8805-414b2fb701c0","Type":"ContainerDied","Data":"c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba"} Jul 08 02:52:33.179346 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.178901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f7768d9d9-cpvjk" event={"ID":"2bfd2fb4-1987-4e98-8805-414b2fb701c0","Type":"ContainerDied","Data":"77de1a02122ea38e3ae0620707dd3106008a36935d26b86fa07140e8a4633502"} Jul 08 02:52:33.179346 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.178925 2576 scope.go:117] "RemoveContainer" containerID="c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba" Jul 08 02:52:33.188033 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.188017 2576 scope.go:117] "RemoveContainer" containerID="c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba" Jul 08 02:52:33.188309 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:52:33.188285 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba\": container with ID starting with c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba not found: ID does not exist" containerID="c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba" Jul 08 02:52:33.188406 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.188321 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba"} err="failed to get container status \"c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba\": rpc error: code = NotFound desc = could not find container \"c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba\": container with ID starting with c8f61f45fb06bfe90f37b29d14cb9b680b4dbd8a858764b4730a3a5f597646ba not found: ID does not exist" Jul 08 02:52:33.200962 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.200937 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f7768d9d9-cpvjk"] Jul 08 02:52:33.207619 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:33.207600 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f7768d9d9-cpvjk"] Jul 08 02:52:34.278152 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:34.278118 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bfd2fb4-1987-4e98-8805-414b2fb701c0" path="/var/lib/kubelet/pods/2bfd2fb4-1987-4e98-8805-414b2fb701c0/volumes" Jul 08 02:52:50.981895 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:50.981840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:51.404289 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:51.404254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:51.786524 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:51.786444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:51.886955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:51.886929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:52:51.960699 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:52:51.960669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:53:01.629620 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:53:01.629586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:53:01.735603 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:53:01.735571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:53:07.516244 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:53:07.516212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:53:18.862497 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:53:18.862453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:53:38.728156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:53:38.728123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:10.451701 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:10.451671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:24.450735 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.450703 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2"] Jul 08 02:54:24.451140 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.451093 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfd2fb4-1987-4e98-8805-414b2fb701c0" containerName="manager" Jul 08 02:54:24.451140 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.451106 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfd2fb4-1987-4e98-8805-414b2fb701c0" containerName="manager" Jul 08 02:54:24.451223 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.451176 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfd2fb4-1987-4e98-8805-414b2fb701c0" containerName="manager" Jul 08 02:54:24.454279 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.454263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.458172 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.458147 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4sql4\"" Jul 08 02:54:24.458172 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.458165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 02:54:24.458390 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.458176 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 02:54:24.458390 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.458234 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 02:54:24.465371 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.465327 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2"] Jul 08 02:54:24.480387 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.480365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:24.619133 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.619108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.619261 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.619136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.619261 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.619172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbks\" (UniqueName: \"kubernetes.io/projected/89c16bbc-ca04-4941-bc61-b672e51b02dc-kube-api-access-bnbks\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.619340 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.619249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c16bbc-ca04-4941-bc61-b672e51b02dc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.619340 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.619302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.619340 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.619325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.720621 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.720550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbks\" (UniqueName: \"kubernetes.io/projected/89c16bbc-ca04-4941-bc61-b672e51b02dc-kube-api-access-bnbks\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.720621 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.720600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c16bbc-ca04-4941-bc61-b672e51b02dc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.720838 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.720626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.720838 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.720642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.720838 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.720695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.720838 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.720712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.721084 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.721057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.721190 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.721135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.721190 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.721138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.723076 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.723053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c16bbc-ca04-4941-bc61-b672e51b02dc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.723186 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.723170 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c16bbc-ca04-4941-bc61-b672e51b02dc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.730033 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.729994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbks\" (UniqueName: \"kubernetes.io/projected/89c16bbc-ca04-4941-bc61-b672e51b02dc-kube-api-access-bnbks\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2\" (UID: \"89c16bbc-ca04-4941-bc61-b672e51b02dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.764766 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.764738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:24.900601 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:24.900572 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2"] Jul 08 02:54:24.904119 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:54:24.904084 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c16bbc_ca04_4941_bc61_b672e51b02dc.slice/crio-fa8aafe48c26d1fd9fb02f0d8ee82abc3dca0f9530223a9358ce40345408edb4 WatchSource:0}: Error finding container fa8aafe48c26d1fd9fb02f0d8ee82abc3dca0f9530223a9358ce40345408edb4: Status 404 returned error can't find the container with id fa8aafe48c26d1fd9fb02f0d8ee82abc3dca0f9530223a9358ce40345408edb4 Jul 08 02:54:25.680080 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:25.680037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" event={"ID":"89c16bbc-ca04-4941-bc61-b672e51b02dc","Type":"ContainerStarted","Data":"fa8aafe48c26d1fd9fb02f0d8ee82abc3dca0f9530223a9358ce40345408edb4"} Jul 08 02:54:26.686259 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:26.686219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" event={"ID":"89c16bbc-ca04-4941-bc61-b672e51b02dc","Type":"ContainerStarted","Data":"81cfb292a172b71e98639554728e6214a6fc05fb5e801a3fd3c0c27120ee9f2e"} Jul 08 02:54:26.704220 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:26.704171 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" podStartSLOduration=1.692666307 podStartE2EDuration="2.704154919s" totalImagesPullingTime="1.011488612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:54:26.703827404 +0000 UTC m=+856.982204633" watchObservedRunningTime="2026-07-08 02:54:26.704154919 +0000 UTC m=+856.982532124" Jul 08 02:54:34.764920 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:34.764886 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:34.765414 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:34.764940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:34.777387 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:34.777363 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:35.743751 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:35.743728 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2" Jul 08 02:54:36.081982 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:36.081915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:38.992817 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:38.992780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:46.782011 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:46.781983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:49.066249 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:49.066213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:59.534263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:59.534227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:59.590376 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:59.590319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:54:59.814851 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:54:59.814769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:55:10.173648 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:10.173567 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:55:10.177960 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:10.177937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 02:55:17.465549 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:17.465518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:55:18.558770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:18.558742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:55:46.916575 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:46.916544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:55:47.012977 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:47.012949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:55:48.049090 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:48.049055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:55:48.140153 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:55:48.140125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:00.622788 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:00.622752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:00.707233 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:00.707203 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664c944698-mqxlj"] Jul 08 02:57:00.707486 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:00.707426 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-664c944698-mqxlj" podUID="c830238e-9adb-4338-8377-08073ff115d3" containerName="manager" containerID="cri-o://9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b" gracePeriod=30 Jul 08 02:57:00.947541 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:00.947517 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:57:01.076671 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.076645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c830238e-9adb-4338-8377-08073ff115d3-service-ca\") pod \"c830238e-9adb-4338-8377-08073ff115d3\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " Jul 08 02:57:01.076840 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.076732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c830238e-9adb-4338-8377-08073ff115d3-webhook-cert\") pod \"c830238e-9adb-4338-8377-08073ff115d3\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " Jul 08 02:57:01.076840 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.076769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2vpm\" (UniqueName: \"kubernetes.io/projected/c830238e-9adb-4338-8377-08073ff115d3-kube-api-access-b2vpm\") pod \"c830238e-9adb-4338-8377-08073ff115d3\" (UID: \"c830238e-9adb-4338-8377-08073ff115d3\") " Jul 08 02:57:01.077018 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.076981 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c830238e-9adb-4338-8377-08073ff115d3-service-ca" (OuterVolumeSpecName: "service-ca") pod "c830238e-9adb-4338-8377-08073ff115d3" (UID: "c830238e-9adb-4338-8377-08073ff115d3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:57:01.078793 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.078768 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c830238e-9adb-4338-8377-08073ff115d3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c830238e-9adb-4338-8377-08073ff115d3" (UID: "c830238e-9adb-4338-8377-08073ff115d3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:57:01.079112 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.079093 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c830238e-9adb-4338-8377-08073ff115d3-kube-api-access-b2vpm" (OuterVolumeSpecName: "kube-api-access-b2vpm") pod "c830238e-9adb-4338-8377-08073ff115d3" (UID: "c830238e-9adb-4338-8377-08073ff115d3"). InnerVolumeSpecName "kube-api-access-b2vpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:57:01.177680 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.177652 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vpm\" (UniqueName: \"kubernetes.io/projected/c830238e-9adb-4338-8377-08073ff115d3-kube-api-access-b2vpm\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:57:01.177680 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.177677 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c830238e-9adb-4338-8377-08073ff115d3-service-ca\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:57:01.177834 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.177686 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c830238e-9adb-4338-8377-08073ff115d3-webhook-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 02:57:01.375516 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.375493 2576 generic.go:356] "Generic (PLEG): container finished" podID="c830238e-9adb-4338-8377-08073ff115d3" containerID="9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b" exitCode=0 Jul 08 02:57:01.375628 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.375521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664c944698-mqxlj" event={"ID":"c830238e-9adb-4338-8377-08073ff115d3","Type":"ContainerDied","Data":"9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b"} Jul 08 02:57:01.375628 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.375546 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664c944698-mqxlj" Jul 08 02:57:01.375628 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.375568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664c944698-mqxlj" event={"ID":"c830238e-9adb-4338-8377-08073ff115d3","Type":"ContainerDied","Data":"ee9ef43527025e2907163bbf86d9dca01c9c2c9b4350ae66fe03a7162918e99f"} Jul 08 02:57:01.375628 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.375595 2576 scope.go:117] "RemoveContainer" containerID="9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b" Jul 08 02:57:01.384259 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.384241 2576 scope.go:117] "RemoveContainer" containerID="9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b" Jul 08 02:57:01.384558 ip-10-0-128-107 kubenswrapper[2576]: E0708 02:57:01.384539 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b\": container with ID starting with 9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b not found: ID does not exist" containerID="9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b" Jul 08 02:57:01.384612 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.384565 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b"} err="failed to get container status \"9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b\": rpc error: code = NotFound desc = could not find container \"9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b\": container with ID starting with 9272afd6b7323aac6b989e1a7e423aee214d38b609be8a8bcbef9821972b4a6b not found: ID does not exist" Jul 08 02:57:01.397263 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.397238 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664c944698-mqxlj"] Jul 08 02:57:01.401060 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:01.401038 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-664c944698-mqxlj"] Jul 08 02:57:02.103702 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.103672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:02.223770 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.223739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-664c944698-jhg88"] Jul 08 02:57:02.224077 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.224066 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c830238e-9adb-4338-8377-08073ff115d3" containerName="manager" Jul 08 02:57:02.224128 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.224079 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c830238e-9adb-4338-8377-08073ff115d3" containerName="manager" Jul 08 02:57:02.224169 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.224143 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c830238e-9adb-4338-8377-08073ff115d3" containerName="manager" Jul 08 02:57:02.228198 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.228177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.231862 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.231845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 02:57:02.231955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.231866 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ws2\"" Jul 08 02:57:02.231955 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.231869 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 02:57:02.239243 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.239222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664c944698-jhg88"] Jul 08 02:57:02.278465 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.278439 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c830238e-9adb-4338-8377-08073ff115d3" path="/var/lib/kubelet/pods/c830238e-9adb-4338-8377-08073ff115d3/volumes" Jul 08 02:57:02.389886 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.389859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-service-ca\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.390024 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.389897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-webhook-cert\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.390024 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.389932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njmr\" (UniqueName: \"kubernetes.io/projected/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-kube-api-access-4njmr\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.491247 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.491223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-service-ca\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.491348 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.491255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-webhook-cert\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.491348 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.491287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njmr\" (UniqueName: \"kubernetes.io/projected/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-kube-api-access-4njmr\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.491837 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.491820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-service-ca\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.493730 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.493713 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-webhook-cert\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.500987 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.500966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njmr\" (UniqueName: \"kubernetes.io/projected/f7187e5d-e3df-4c6c-8dfe-60fbab125d9b-kube-api-access-4njmr\") pod \"maas-controller-664c944698-jhg88\" (UID: \"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b\") " pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.538075 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.538043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:02.673012 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.672923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664c944698-jhg88"] Jul 08 02:57:02.676110 ip-10-0-128-107 kubenswrapper[2576]: W0708 02:57:02.676086 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7187e5d_e3df_4c6c_8dfe_60fbab125d9b.slice/crio-4eb36097e3b2e8b79b2f68e40f5bc8db6bf2d3a49eb1c9fb628387997de5dc40 WatchSource:0}: Error finding container 4eb36097e3b2e8b79b2f68e40f5bc8db6bf2d3a49eb1c9fb628387997de5dc40: Status 404 returned error can't find the container with id 4eb36097e3b2e8b79b2f68e40f5bc8db6bf2d3a49eb1c9fb628387997de5dc40 Jul 08 02:57:02.677883 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:02.677864 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:57:03.388137 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:03.388103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664c944698-jhg88" event={"ID":"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b","Type":"ContainerStarted","Data":"4eb36097e3b2e8b79b2f68e40f5bc8db6bf2d3a49eb1c9fb628387997de5dc40"} Jul 08 02:57:04.393156 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:04.393116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664c944698-jhg88" event={"ID":"f7187e5d-e3df-4c6c-8dfe-60fbab125d9b","Type":"ContainerStarted","Data":"012b080bb15990878e50122041e93069fc5fc35f2ef30c263d8e86e79e217fff"} Jul 08 02:57:04.393623 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:04.393230 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:04.411683 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:04.411640 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-664c944698-jhg88" podStartSLOduration=1.475146873 podStartE2EDuration="2.411626129s" totalImagesPullingTime="936.479256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:57:04.410231271 +0000 UTC m=+1014.688608483" watchObservedRunningTime="2026-07-08 02:57:04.411626129 +0000 UTC m=+1014.690003350" Jul 08 02:57:15.402555 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:15.402524 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-664c944698-jhg88" Jul 08 02:57:44.018195 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:44.018129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:44.109033 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:44.109005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:44.225953 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:44.225909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:50.145661 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:50.145625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:50.253712 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:50.253684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:50.346547 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:50.346520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:50.497757 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:50.497660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:56.142640 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:56.142606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:57:56.243938 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:57:56.243910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:20.115554 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:59:20.115522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:50.314864 ip-10-0-128-107 kubenswrapper[2576]: I0708 02:59:50.314834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:10.206982 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:00:10.206957 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:00:10.211028 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:00:10.211005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:00:14.720770 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:00:14.720743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:14.820665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:00:14.820635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:03:33.921818 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:03:33.921772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:03:35.159071 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:03:35.159042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:03:45.691763 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:03:45.691721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:10.235061 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:05:10.235036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:05:10.239820 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:05:10.239795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:05:53.746570 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:05:53.746530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:43.085531 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:07:43.085504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:38.280471 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:08:38.280410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:54.310917 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:08:54.310885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:02.571530 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:02.571490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:07.685506 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:07.685470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:10.546052 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:10.546020 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:10.643894 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:10.643859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:11.038058 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:11.038030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:16.902103 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:16.902069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:17.002445 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.002413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:17.089827 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.089798 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np"] Jul 08 03:09:17.094007 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.093983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.096588 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.096572 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15fb6db9-openshift-default-dockercfg-8csfj\"" Jul 08 03:09:17.114024 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.114002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np"] Jul 08 03:09:17.123310 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.123290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:17.133169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-certs\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133264 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-envoy\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133264 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-podinfo\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133379 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89jg\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-kube-api-access-b89jg\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133379 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-data\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133379 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-token\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133488 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istiod-ca-cert\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133488 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-socket\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.133488 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.133444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-credential-socket\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.224472 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.224399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:17.234446 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-token\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234534 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istiod-ca-cert\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234534 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-socket\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234534 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-credential-socket\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234701 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-certs\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234701 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-envoy\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234701 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-podinfo\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234701 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89jg\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-kube-api-access-b89jg\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234886 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-data\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.234940 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.234900 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-credential-socket\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.235082 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.235039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-certs\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.235187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.235113 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-data\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.235187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.235138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-socket\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.235289 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.235271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istiod-ca-cert\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.236973 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.236947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-envoy\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.237090 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.237071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-podinfo\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.243418 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.243400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-token\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.243667 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.243649 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89jg\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-kube-api-access-b89jg\") pod \"e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.405642 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.405616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:17.542823 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.542797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np"] Jul 08 03:09:17.544582 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:09:17.544561 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfb4e0b_299d_465a_87f9_f8e8eb8c9de8.slice/crio-d231b5b9a12d6ed4801a8fb36a40a6276d2b5efbf96007422ce1ca961bd5bb20 WatchSource:0}: Error finding container d231b5b9a12d6ed4801a8fb36a40a6276d2b5efbf96007422ce1ca961bd5bb20: Status 404 returned error can't find the container with id d231b5b9a12d6ed4801a8fb36a40a6276d2b5efbf96007422ce1ca961bd5bb20 Jul 08 03:09:17.546636 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.546607 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:17.546711 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.546665 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:17.546711 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.546692 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:17.624480 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.624457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" event={"ID":"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8","Type":"ContainerStarted","Data":"7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a"} Jul 08 03:09:17.624574 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.624501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" event={"ID":"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8","Type":"ContainerStarted","Data":"d231b5b9a12d6ed4801a8fb36a40a6276d2b5efbf96007422ce1ca961bd5bb20"} Jul 08 03:09:17.648659 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:17.648607 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" podStartSLOduration=0.648587287 podStartE2EDuration="648.587287ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:09:17.645855808 +0000 UTC m=+1747.924233008" watchObservedRunningTime="2026-07-08 03:09:17.648587287 +0000 UTC m=+1747.926964489" Jul 08 03:09:18.304587 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.304535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:18.404580 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.404551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:18.406490 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.406474 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:18.410658 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.410636 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:18.505931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.505897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:18.629155 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.629131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:18.630117 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.630099 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:18.873439 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.873402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:18.978199 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:18.978108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:19.074227 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:19.074188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:20.745769 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:20.745738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:26.358109 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:26.358079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:26.417066 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:26.417041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:26.516595 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:26.516523 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np"] Jul 08 03:09:26.516795 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:26.516771 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" podUID="4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" containerName="istio-proxy" containerID="cri-o://7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a" gracePeriod=30 Jul 08 03:09:26.541218 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:26.541189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:26.664658 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:26.664625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:28.510541 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:28.510500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:29.270977 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.270955 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:29.327394 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327330 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-token\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327394 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-socket\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327529 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-data\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327529 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327500 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-envoy\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327529 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327526 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-credential-socket\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327643 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327553 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-podinfo\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327643 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istiod-ca-cert\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327643 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327603 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-certs\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327643 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327637 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b89jg\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-kube-api-access-b89jg\") pod \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\" (UID: \"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8\") " Jul 08 03:09:29.327833 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327687 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:29.327833 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327795 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:29.328006 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327966 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.328006 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.327992 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.328157 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.328011 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:09:29.328157 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.328031 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:29.328580 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.328554 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:29.329772 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.329748 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:29.329970 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.329952 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:09:29.330051 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.329963 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:09:29.330386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.330338 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-kube-api-access-b89jg" (OuterVolumeSpecName: "kube-api-access-b89jg") pod "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" (UID: "4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8"). InnerVolumeSpecName "kube-api-access-b89jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:09:29.428497 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428466 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.428497 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428496 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.428497 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428505 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.428659 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428515 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.428659 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428523 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89jg\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-kube-api-access-b89jg\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.428659 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428532 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.428659 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.428539 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:29.681103 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.681073 2576 generic.go:356] "Generic (PLEG): container finished" podID="4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" containerID="7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a" exitCode=0 Jul 08 03:09:29.681452 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.681117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" event={"ID":"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8","Type":"ContainerDied","Data":"7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a"} Jul 08 03:09:29.681452 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.681123 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" Jul 08 03:09:29.681452 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.681167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np" event={"ID":"4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8","Type":"ContainerDied","Data":"d231b5b9a12d6ed4801a8fb36a40a6276d2b5efbf96007422ce1ca961bd5bb20"} Jul 08 03:09:29.681452 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.681198 2576 scope.go:117] "RemoveContainer" containerID="7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a" Jul 08 03:09:29.690338 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.690323 2576 scope.go:117] "RemoveContainer" containerID="7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a" Jul 08 03:09:29.690604 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:09:29.690585 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a\": container with ID starting with 7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a not found: ID does not exist" containerID="7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a" Jul 08 03:09:29.690665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.690613 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a"} err="failed to get container status \"7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a\": rpc error: code = NotFound desc = could not find container \"7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a\": container with ID starting with 7537f7a94eedfee3a53864c4eab7a541fe044eef9fee7a9037997728cc970f3a not found: ID does not exist" Jul 08 03:09:29.702814 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.702794 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np"] Jul 08 03:09:29.707894 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:29.707873 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15fb6db9-openshift-default-7f4bf75fb5-j59np"] Jul 08 03:09:30.279039 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:30.279005 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" path="/var/lib/kubelet/pods/4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8/volumes" Jul 08 03:09:32.618545 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.618516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:32.720313 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.720284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:32.793223 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.793186 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h"] Jul 08 03:09:32.793714 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.793689 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" containerName="istio-proxy" Jul 08 03:09:32.793846 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.793779 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" containerName="istio-proxy" Jul 08 03:09:32.793982 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.793956 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dfb4e0b-299d-465a-87f9-f8e8eb8c9de8" containerName="istio-proxy" Jul 08 03:09:32.797446 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.797423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.800397 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.800374 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-32f16294-openshift-default-dockercfg-txsnx\"" Jul 08 03:09:32.810427 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.810405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h"] Jul 08 03:09:32.840513 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.840487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:32.852732 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.852706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n8p\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-kube-api-access-45n8p\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.852839 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.852749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-certs\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.852839 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.852812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-socket\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.852948 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.852844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-istio-token\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.852948 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.852888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-envoy\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.853050 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.852948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-credential-socket\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.853050 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.853023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f94c856-2d72-4be4-9938-42094adc81e0-istiod-ca-cert\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.853050 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.853045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-data\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.853169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.853068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f94c856-2d72-4be4-9938-42094adc81e0-istio-podinfo\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.931774 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.931736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:32.954297 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-credential-socket\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954451 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f94c856-2d72-4be4-9938-42094adc81e0-istiod-ca-cert\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954451 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-data\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954451 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f94c856-2d72-4be4-9938-42094adc81e0-istio-podinfo\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954596 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n8p\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-kube-api-access-45n8p\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954652 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-certs\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954652 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-socket\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954760 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-istio-token\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954825 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-credential-socket\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954889 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-envoy\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.954946 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954912 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-certs\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.955000 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.954971 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-socket\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.955067 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.955044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-data\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.955183 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.955161 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f94c856-2d72-4be4-9938-42094adc81e0-istiod-ca-cert\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.956784 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.956759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f94c856-2d72-4be4-9938-42094adc81e0-istio-podinfo\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.956867 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.956825 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-envoy\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.965455 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.965435 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-istio-token\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:32.966156 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:32.966135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n8p\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-kube-api-access-45n8p\") pod \"e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:33.109976 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.109951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:33.245336 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.245312 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h"] Jul 08 03:09:33.247742 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:09:33.247714 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f94c856_2d72_4be4_9938_42094adc81e0.slice/crio-1c735fd44ca4021e8cafdf352da8ae512d4441bd2f5eb01a013615ff091dac8e WatchSource:0}: Error finding container 1c735fd44ca4021e8cafdf352da8ae512d4441bd2f5eb01a013615ff091dac8e: Status 404 returned error can't find the container with id 1c735fd44ca4021e8cafdf352da8ae512d4441bd2f5eb01a013615ff091dac8e Jul 08 03:09:33.249853 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.249822 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:33.249929 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.249885 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:33.249929 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.249912 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:33.699680 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.699652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" event={"ID":"2f94c856-2d72-4be4-9938-42094adc81e0","Type":"ContainerStarted","Data":"9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50"} Jul 08 03:09:33.700038 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.699699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" event={"ID":"2f94c856-2d72-4be4-9938-42094adc81e0","Type":"ContainerStarted","Data":"1c735fd44ca4021e8cafdf352da8ae512d4441bd2f5eb01a013615ff091dac8e"} Jul 08 03:09:33.720303 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.720252 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" podStartSLOduration=1.7202319799999999 podStartE2EDuration="1.72023198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:09:33.719729944 +0000 UTC m=+1763.998107175" watchObservedRunningTime="2026-07-08 03:09:33.72023198 +0000 UTC m=+1763.998609183" Jul 08 03:09:33.944061 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:33.944032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:34.045755 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.045689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:34.111104 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.111080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:34.115895 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.115869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:34.116002 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.115917 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:34.215644 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.215615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:34.616893 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.616867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:34.704732 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.704709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:34.705704 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.705688 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:34.721121 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:34.721095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:36.134567 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:36.134537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:42.077243 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:42.077211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:42.133412 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:42.133382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:43.022901 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:43.022865 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h"] Jul 08 03:09:43.023088 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:43.023044 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" podUID="2f94c856-2d72-4be4-9938-42094adc81e0" containerName="istio-proxy" containerID="cri-o://9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50" gracePeriod=30 Jul 08 03:09:43.046140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:43.046087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:43.143693 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:43.143658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:46.418139 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:46.418046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:48.926644 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:48.926610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:49.026905 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.026875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:49.092696 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.090136 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd"] Jul 08 03:09:49.093990 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.093960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.096820 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.096794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f1ce6bf4-openshift-default-dockercfg-klflf\"" Jul 08 03:09:49.107603 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.107581 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd"] Jul 08 03:09:49.145367 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.145330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:49.188636 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-socket\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188636 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdzx\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-kube-api-access-nfdzx\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188776 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istiod-ca-cert\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188776 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-podinfo\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188776 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-token\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188875 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-certs\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188875 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-envoy\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188938 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-credential-socket\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.188972 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.188942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-data\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.259479 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.259446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:49.290291 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-credential-socket\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290437 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-data\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290437 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-socket\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290437 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdzx\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-kube-api-access-nfdzx\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290437 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istiod-ca-cert\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290641 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-podinfo\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290641 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-token\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290641 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-certs\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290641 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-envoy\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290836 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-credential-socket\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290836 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-socket\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.290836 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.290802 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-data\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.291036 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.291014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-certs\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.291253 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.291231 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istiod-ca-cert\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.292967 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.292943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-envoy\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.293197 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.293179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-podinfo\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.306887 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.306862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdzx\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-kube-api-access-nfdzx\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.310217 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.310188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-token\") pod \"e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.406184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.406145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:49.575405 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.575378 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd"] Jul 08 03:09:49.576855 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:09:49.576827 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c09812_5bef_4ada_9ba6_7c9274b85a92.slice/crio-7924abf47123d292cc2e927ad31fe0e616eb87942fa01ca89cf1baefaf92ae56 WatchSource:0}: Error finding container 7924abf47123d292cc2e927ad31fe0e616eb87942fa01ca89cf1baefaf92ae56: Status 404 returned error can't find the container with id 7924abf47123d292cc2e927ad31fe0e616eb87942fa01ca89cf1baefaf92ae56 Jul 08 03:09:49.578972 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.578940 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:49.579036 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.579001 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:49.579036 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.579031 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:09:49.773296 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.773198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" event={"ID":"f8c09812-5bef-4ada-9ba6-7c9274b85a92","Type":"ContainerStarted","Data":"541eabe8f686e118e3e98b18daf9b552798808d7950f2b06d84ebf55e564838b"} Jul 08 03:09:49.773296 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.773260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" event={"ID":"f8c09812-5bef-4ada-9ba6-7c9274b85a92","Type":"ContainerStarted","Data":"7924abf47123d292cc2e927ad31fe0e616eb87942fa01ca89cf1baefaf92ae56"} Jul 08 03:09:49.796220 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.796134 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" podStartSLOduration=0.796115621 podStartE2EDuration="796.115621ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:09:49.794527411 +0000 UTC m=+1780.072904611" watchObservedRunningTime="2026-07-08 03:09:49.796115621 +0000 UTC m=+1780.074492822" Jul 08 03:09:49.979401 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.978563 2576 patch_prober.go:28] interesting pod/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:36876->10.133.0.36:15021: read: connection reset by peer" start-of-body= Jul 08 03:09:49.979401 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:49.978626 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" podUID="2f94c856-2d72-4be4-9938-42094adc81e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:36876->10.133.0.36:15021: read: connection reset by peer" Jul 08 03:09:50.160843 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.160821 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:50.298137 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298032 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-credential-socket\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298137 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f94c856-2d72-4be4-9938-42094adc81e0-istiod-ca-cert\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298128 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-istio-token\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298389 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f94c856-2d72-4be4-9938-42094adc81e0-istio-podinfo\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45n8p\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-kube-api-access-45n8p\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-data\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298504 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-envoy\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-socket\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.298661 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298587 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-certs\") pod \"2f94c856-2d72-4be4-9938-42094adc81e0\" (UID: \"2f94c856-2d72-4be4-9938-42094adc81e0\") " Jul 08 03:09:50.299058 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298722 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f94c856-2d72-4be4-9938-42094adc81e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:09:50.299058 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298954 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f94c856-2d72-4be4-9938-42094adc81e0-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.299058 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.298998 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:50.299300 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.299136 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:50.299517 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.299493 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:50.299702 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.299659 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:50.301652 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.301135 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:09:50.301652 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.301616 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-kube-api-access-45n8p" (OuterVolumeSpecName: "kube-api-access-45n8p") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "kube-api-access-45n8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:09:50.301862 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.301708 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:09:50.301862 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.301830 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f94c856-2d72-4be4-9938-42094adc81e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f94c856-2d72-4be4-9938-42094adc81e0" (UID: "2f94c856-2d72-4be4-9938-42094adc81e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:09:50.332560 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.332537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:50.400003 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.399979 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400003 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400004 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f94c856-2d72-4be4-9938-42094adc81e0-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400204 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400013 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45n8p\" (UniqueName: \"kubernetes.io/projected/2f94c856-2d72-4be4-9938-42094adc81e0-kube-api-access-45n8p\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400204 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400023 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400204 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400033 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400204 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400041 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400204 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400048 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.400204 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.400056 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f94c856-2d72-4be4-9938-42094adc81e0-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:09:50.406851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.406827 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:50.412270 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.412250 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:50.418798 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.418778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:50.499725 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.499702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:50.756806 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.756769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:50.779096 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.779068 2576 generic.go:356] "Generic (PLEG): container finished" podID="2f94c856-2d72-4be4-9938-42094adc81e0" containerID="9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50" exitCode=0 Jul 08 03:09:50.779226 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.779119 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" Jul 08 03:09:50.779226 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.779154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" event={"ID":"2f94c856-2d72-4be4-9938-42094adc81e0","Type":"ContainerDied","Data":"9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50"} Jul 08 03:09:50.779226 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.779213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h" event={"ID":"2f94c856-2d72-4be4-9938-42094adc81e0","Type":"ContainerDied","Data":"1c735fd44ca4021e8cafdf352da8ae512d4441bd2f5eb01a013615ff091dac8e"} Jul 08 03:09:50.779348 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.779248 2576 scope.go:117] "RemoveContainer" containerID="9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50" Jul 08 03:09:50.779615 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.779585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:50.780672 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.780555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:09:50.788210 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.788178 2576 scope.go:117] "RemoveContainer" containerID="9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50" Jul 08 03:09:50.788458 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:09:50.788434 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50\": container with ID starting with 9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50 not found: ID does not exist" containerID="9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50" Jul 08 03:09:50.788511 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.788468 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50"} err="failed to get container status \"9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50\": rpc error: code = NotFound desc = could not find container \"9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50\": container with ID starting with 9d5f8f968c353cd6c6e4950eada157f170f80ce563a6083ac0a3eccc7f075f50 not found: ID does not exist" Jul 08 03:09:50.832793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.832766 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h"] Jul 08 03:09:50.839974 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:50.839950 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-32f16294-openshift-default-6c5594c9b5-5cw2h"] Jul 08 03:09:51.096770 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:51.096698 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:51.198597 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:51.198572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:52.278628 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:52.278594 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f94c856-2d72-4be4-9938-42094adc81e0" path="/var/lib/kubelet/pods/2f94c856-2d72-4be4-9938-42094adc81e0/volumes" Jul 08 03:09:52.424988 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:52.424953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:55.293561 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:55.293526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:55.349791 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:55.349744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:57.671459 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:57.671425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:57.768039 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:57.768005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:57.815104 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:57.815072 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd"] Jul 08 03:09:57.815665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:57.815636 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" podUID="f8c09812-5bef-4ada-9ba6-7c9274b85a92" containerName="istio-proxy" containerID="cri-o://541eabe8f686e118e3e98b18daf9b552798808d7950f2b06d84ebf55e564838b" gracePeriod=30 Jul 08 03:09:57.866423 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:57.866392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:57.966394 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:57.966304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:59.320235 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:09:59.320185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:01.842586 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:01.842515 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8c09812-5bef-4ada-9ba6-7c9274b85a92" containerID="541eabe8f686e118e3e98b18daf9b552798808d7950f2b06d84ebf55e564838b" exitCode=0 Jul 08 03:10:01.842586 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:01.842556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" event={"ID":"f8c09812-5bef-4ada-9ba6-7c9274b85a92","Type":"ContainerDied","Data":"541eabe8f686e118e3e98b18daf9b552798808d7950f2b06d84ebf55e564838b"} Jul 08 03:10:01.871989 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:01.871971 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:10:02.001106 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001050 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-credential-socket\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001106 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001076 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-data\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001106 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001099 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-podinfo\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-certs\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001136 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfdzx\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-kube-api-access-nfdzx\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001152 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istiod-ca-cert\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001256 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-socket\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001276 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-token\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-envoy\") pod \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\" (UID: \"f8c09812-5bef-4ada-9ba6-7c9274b85a92\") " Jul 08 03:10:02.001799 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001462 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:10:02.001799 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001553 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:10:02.001799 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001673 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.001799 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001691 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.001799 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001686 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:10:02.001799 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001746 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:10:02.002051 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.001915 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:10:02.003629 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.003605 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:10:02.003719 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.003636 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:10:02.003888 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.003875 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:10:02.003983 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.003969 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-kube-api-access-nfdzx" (OuterVolumeSpecName: "kube-api-access-nfdzx") pod "f8c09812-5bef-4ada-9ba6-7c9274b85a92" (UID: "f8c09812-5bef-4ada-9ba6-7c9274b85a92"). InnerVolumeSpecName "kube-api-access-nfdzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:10:02.102180 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102157 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.102180 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102180 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.102312 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102189 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.102312 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102197 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.102312 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102204 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdzx\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-kube-api-access-nfdzx\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.102312 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102213 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.102312 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.102220 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c09812-5bef-4ada-9ba6-7c9274b85a92-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:10:02.848645 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.848617 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" Jul 08 03:10:02.848645 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.848630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd" event={"ID":"f8c09812-5bef-4ada-9ba6-7c9274b85a92","Type":"ContainerDied","Data":"7924abf47123d292cc2e927ad31fe0e616eb87942fa01ca89cf1baefaf92ae56"} Jul 08 03:10:02.849119 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.848692 2576 scope.go:117] "RemoveContainer" containerID="541eabe8f686e118e3e98b18daf9b552798808d7950f2b06d84ebf55e564838b" Jul 08 03:10:02.866955 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.866935 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd"] Jul 08 03:10:02.870428 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:02.870409 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f1ce6bf4-openshift-default-d6cc6667-gdnpd"] Jul 08 03:10:03.685276 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:03.685248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:03.785766 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:03.785737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:04.149396 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:04.149343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:04.244707 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:04.244681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:04.278438 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:04.278416 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c09812-5bef-4ada-9ba6-7c9274b85a92" path="/var/lib/kubelet/pods/f8c09812-5bef-4ada-9ba6-7c9274b85a92/volumes" Jul 08 03:10:04.348788 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:04.348766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:10.262920 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:10.262891 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:10:10.270140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:10.270122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:10:35.656442 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:35.656408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:41.619817 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:41.619768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:47.278278 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:47.278233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:47.451187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:47.451154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:47.528050 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:47.528025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:48.015960 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:10:48.015935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:13.996376 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:13.996315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:19.976960 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:19.976925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:20.082932 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:20.082895 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:20.177005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:20.176973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:36.084416 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:36.084384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:41.775286 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:41.775245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:41.852083 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:41.852056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:41.980910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:41.980884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:42.087939 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:42.087876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:52.534652 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:52.534619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:53.328271 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:53.328235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:59.048559 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:59.048523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:59.125909 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:59.125881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:59.210068 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:59.210036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:59.307904 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:59.307837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:59.407218 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:11:59.407193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:00.144365 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:00.144329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:00.239445 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:00.239421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:00.350984 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:00.350953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:26.018693 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:26.018657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:26.119931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:26.119897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:31.863907 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:31.863869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:38.298097 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:38.298061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:44.391532 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:44.391499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:44.535386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:12:44.535338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:15.024069 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:15.023998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:20.690983 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:20.690945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:34.717665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:34.717631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:34.796820 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:34.796790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:34.953784 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:34.953758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:35.288892 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.288862 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6"] Jul 08 03:13:35.289387 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.289370 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f94c856-2d72-4be4-9938-42094adc81e0" containerName="istio-proxy" Jul 08 03:13:35.289465 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.289389 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f94c856-2d72-4be4-9938-42094adc81e0" containerName="istio-proxy" Jul 08 03:13:35.289465 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.289418 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c09812-5bef-4ada-9ba6-7c9274b85a92" containerName="istio-proxy" Jul 08 03:13:35.289465 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.289428 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c09812-5bef-4ada-9ba6-7c9274b85a92" containerName="istio-proxy" Jul 08 03:13:35.289583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.289513 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c09812-5bef-4ada-9ba6-7c9274b85a92" containerName="istio-proxy" Jul 08 03:13:35.289583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.289530 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f94c856-2d72-4be4-9938-42094adc81e0" containerName="istio-proxy" Jul 08 03:13:35.291725 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.291704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.294797 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.294772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50f1f9f8-openshift-default-dockercfg-lbxqj\"" Jul 08 03:13:35.314423 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.314342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6"] Jul 08 03:13:35.315981 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.315960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:35.457776 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-envoy\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.457776 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-data\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wl2\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-kube-api-access-94wl2\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-podinfo\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-certs\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-socket\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-credential-socket\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.457965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-token\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.458299 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.458013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istiod-ca-cert\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.558863 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-envoy\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.558863 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-data\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.558863 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wl2\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-kube-api-access-94wl2\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559105 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-podinfo\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559105 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-certs\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559105 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-socket\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559105 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.558950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-credential-socket\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559105 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-token\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559105 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istiod-ca-cert\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559418 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559288 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-data\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559475 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559442 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-credential-socket\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559475 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-certs\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559576 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559548 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-socket\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.559863 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.559846 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istiod-ca-cert\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.561509 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.561484 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-envoy\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.561983 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.561967 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-podinfo\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.566950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.566930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-token\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.567131 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.567115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wl2\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-kube-api-access-94wl2\") pod \"e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.601846 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.601823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:35.749199 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.747555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6"] Jul 08 03:13:35.751677 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.751630 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:13:35.751790 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.751771 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:13:35.751876 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.751857 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:13:35.775841 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.775815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" event={"ID":"5cf0675f-f032-4a57-b5d5-d5caeeba67e0","Type":"ContainerStarted","Data":"3d7e2b8f02d10e06413423a8f879e45586d069f3497ce08f2b51ea025dfd083a"} Jul 08 03:13:35.804345 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:35.804224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:36.782284 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:36.782250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" event={"ID":"5cf0675f-f032-4a57-b5d5-d5caeeba67e0","Type":"ContainerStarted","Data":"84b88d85243b95eef855d1d2eef3d0a88fcb2109da4f70119e5925b3ded2f752"} Jul 08 03:13:36.804313 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:36.804265 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" podStartSLOduration=1.804250164 podStartE2EDuration="1.804250164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:13:36.801947583 +0000 UTC m=+2007.080324786" watchObservedRunningTime="2026-07-08 03:13:36.804250164 +0000 UTC m=+2007.082627408" Jul 08 03:13:37.602803 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:37.602767 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:37.607451 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:37.607429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:37.787005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:37.786985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:37.787852 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:37.787837 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:38.761331 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:38.761277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:38.843708 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:38.843661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:38.910931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:38.910900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:39.047903 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:39.047806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:39.337632 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:39.337546 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:39.435931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:39.435896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:42.151140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:42.151099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:43.451766 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:43.451713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:43.953137 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:43.953106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:44.046565 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:44.046539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:44.375871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:44.375842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:50.483716 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:50.483662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:50.585415 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:50.585378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:51.050262 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:51.050226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:51.414290 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:51.414255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:51.513224 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:51.513190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:52.788857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.788831 2576 patch_prober.go:28] interesting pod/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 08 03:13:52.789239 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.788877 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 08 03:13:52.859150 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.859127 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerID="84b88d85243b95eef855d1d2eef3d0a88fcb2109da4f70119e5925b3ded2f752" exitCode=0 Jul 08 03:13:52.859273 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.859193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" event={"ID":"5cf0675f-f032-4a57-b5d5-d5caeeba67e0","Type":"ContainerDied","Data":"84b88d85243b95eef855d1d2eef3d0a88fcb2109da4f70119e5925b3ded2f752"} Jul 08 03:13:52.859565 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.859552 2576 scope.go:117] "RemoveContainer" containerID="84b88d85243b95eef855d1d2eef3d0a88fcb2109da4f70119e5925b3ded2f752" Jul 08 03:13:52.860491 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.860465 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:13:52.860557 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.860535 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:13:52.860594 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:52.860566 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:13:53.866225 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:53.866193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" event={"ID":"5cf0675f-f032-4a57-b5d5-d5caeeba67e0","Type":"ContainerStarted","Data":"29d40630a37adbe10ec2c70f9261b2bd01581df4016c7b2bf06f7ca907cb8a6c"} Jul 08 03:13:54.602931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:54.602896 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:13:54.603439 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:54.603414 2576 patch_prober.go:28] interesting pod/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 08 03:13:54.603620 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:54.603595 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 08 03:13:55.602445 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:55.602405 2576 patch_prober.go:28] interesting pod/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 08 03:13:55.602921 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:55.602468 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 08 03:13:56.603055 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:56.602618 2576 patch_prober.go:28] interesting pod/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 08 03:13:56.603055 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:56.602693 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 08 03:13:57.039851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.039816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:57.141347 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.141302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:57.273288 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.273259 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6"] Jul 08 03:13:57.273488 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.273465 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" containerID="cri-o://29d40630a37adbe10ec2c70f9261b2bd01581df4016c7b2bf06f7ca907cb8a6c" gracePeriod=30 Jul 08 03:13:57.297809 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.297616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:57.639797 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.639732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:57.737187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.737156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:57.865205 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:13:57.865171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:02.907812 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.907787 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerID="29d40630a37adbe10ec2c70f9261b2bd01581df4016c7b2bf06f7ca907cb8a6c" exitCode=0 Jul 08 03:14:02.908184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.907867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" event={"ID":"5cf0675f-f032-4a57-b5d5-d5caeeba67e0","Type":"ContainerDied","Data":"29d40630a37adbe10ec2c70f9261b2bd01581df4016c7b2bf06f7ca907cb8a6c"} Jul 08 03:14:02.908184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.907938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" event={"ID":"5cf0675f-f032-4a57-b5d5-d5caeeba67e0","Type":"ContainerDied","Data":"3d7e2b8f02d10e06413423a8f879e45586d069f3497ce08f2b51ea025dfd083a"} Jul 08 03:14:02.908184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.907966 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d7e2b8f02d10e06413423a8f879e45586d069f3497ce08f2b51ea025dfd083a" Jul 08 03:14:02.908184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.907983 2576 scope.go:117] "RemoveContainer" containerID="84b88d85243b95eef855d1d2eef3d0a88fcb2109da4f70119e5925b3ded2f752" Jul 08 03:14:02.915456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.915438 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:14:02.967311 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967247 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-token\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967460 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967335 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-podinfo\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967460 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967394 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-envoy\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967460 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967410 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-data\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967460 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-socket\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967461 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-credential-socket\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967478 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-certs\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istiod-ca-cert\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94wl2\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-kube-api-access-94wl2\") pod \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\" (UID: \"5cf0675f-f032-4a57-b5d5-d5caeeba67e0\") " Jul 08 03:14:02.967856 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967723 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:14:02.967912 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.967889 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:14:02.968077 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.968039 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:14:02.968253 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.968229 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:14:02.968667 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.968635 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:14:02.969678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.969658 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:14:02.969851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.969831 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-kube-api-access-94wl2" (OuterVolumeSpecName: "kube-api-access-94wl2") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "kube-api-access-94wl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:14:02.969981 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.969959 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:14:02.970080 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:02.970015 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cf0675f-f032-4a57-b5d5-d5caeeba67e0" (UID: "5cf0675f-f032-4a57-b5d5-d5caeeba67e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:14:03.068075 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068054 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068075 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068075 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068083 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068091 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068099 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068106 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068114 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068140 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wl2\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-kube-api-access-94wl2\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.068230 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.068148 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf0675f-f032-4a57-b5d5-d5caeeba67e0-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:14:03.426251 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.426224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:03.530925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.530893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:03.625905 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.625875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:03.787231 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.787166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:03.913426 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.913406 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6" Jul 08 03:14:03.948372 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.948326 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6"] Jul 08 03:14:03.954648 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:03.954622 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50f1f9f8-openshift-default-db787b85f-rnxz6"] Jul 08 03:14:04.277640 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:04.277616 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" path="/var/lib/kubelet/pods/5cf0675f-f032-4a57-b5d5-d5caeeba67e0/volumes" Jul 08 03:14:09.374694 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:09.374655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:09.435762 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:09.435738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:16.162899 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.162871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:16.452772 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.452699 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g"] Jul 08 03:14:16.453200 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.453184 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" Jul 08 03:14:16.453323 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.453202 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" Jul 08 03:14:16.453323 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.453222 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" Jul 08 03:14:16.453323 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.453231 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" Jul 08 03:14:16.453583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.453337 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" Jul 08 03:14:16.453583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.453379 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf0675f-f032-4a57-b5d5-d5caeeba67e0" containerName="istio-proxy" Jul 08 03:14:16.455665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.455646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.458029 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.458010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c99a305e-openshift-default-dockercfg-hdsxd\"" Jul 08 03:14:16.470049 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.470027 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g"] Jul 08 03:14:16.478865 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.478844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:16.566546 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540a6681-39b9-4ed8-998b-d2985cfec180-istiod-ca-cert\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.566678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-socket\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.566678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-istio-token\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.566850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-credential-socket\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.566850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540a6681-39b9-4ed8-998b-d2985cfec180-istio-podinfo\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.566850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-data\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.566850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtgn\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-kube-api-access-xhtgn\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.567014 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-certs\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.567014 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.566879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-envoy\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.579319 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.579286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:16.667993 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.667958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-istio-token\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-credential-socket\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540a6681-39b9-4ed8-998b-d2985cfec180-istio-podinfo\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-data\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtgn\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-kube-api-access-xhtgn\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668448 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-certs\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668448 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-envoy\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668448 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540a6681-39b9-4ed8-998b-d2985cfec180-istiod-ca-cert\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668448 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-socket\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668502 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-credential-socket\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668665 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668626 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-data\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668766 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668686 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-socket\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.668766 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.668716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-certs\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.669469 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.669445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540a6681-39b9-4ed8-998b-d2985cfec180-istiod-ca-cert\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.670656 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.670641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-envoy\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.670875 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.670857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540a6681-39b9-4ed8-998b-d2985cfec180-istio-podinfo\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.685762 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.685730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-istio-token\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.686062 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.686043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtgn\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-kube-api-access-xhtgn\") pod \"e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.766200 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.766147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:16.900547 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.900524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g"] Jul 08 03:14:16.902709 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:14:16.902682 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540a6681_39b9_4ed8_998b_d2985cfec180.slice/crio-0a76a4c3d7fc51f574decaedb0098dfe7b32a8c4d4540e11961c956268c9b89c WatchSource:0}: Error finding container 0a76a4c3d7fc51f574decaedb0098dfe7b32a8c4d4540e11961c956268c9b89c: Status 404 returned error can't find the container with id 0a76a4c3d7fc51f574decaedb0098dfe7b32a8c4d4540e11961c956268c9b89c Jul 08 03:14:16.904754 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.904716 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:14:16.904882 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.904795 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:14:16.904882 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.904844 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:14:16.973943 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.973912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" event={"ID":"540a6681-39b9-4ed8-998b-d2985cfec180","Type":"ContainerStarted","Data":"23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806"} Jul 08 03:14:16.974040 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.973971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" event={"ID":"540a6681-39b9-4ed8-998b-d2985cfec180","Type":"ContainerStarted","Data":"0a76a4c3d7fc51f574decaedb0098dfe7b32a8c4d4540e11961c956268c9b89c"} Jul 08 03:14:16.995934 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:16.995890 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podStartSLOduration=0.995876658 podStartE2EDuration="995.876658ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:14:16.995067604 +0000 UTC m=+2047.273444840" watchObservedRunningTime="2026-07-08 03:14:16.995876658 +0000 UTC m=+2047.274253859" Jul 08 03:14:17.767257 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:17.767225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:17.771672 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:17.771649 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:17.978849 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:17.978825 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:17.979612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:17.979591 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:18.899888 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:18.899853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:18.990267 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:18.990227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:19.054589 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:19.054562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:19.154002 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:19.153935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:19.568094 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:19.568002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:19.668919 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:19.668892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:23.401415 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:23.401377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:33.162370 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:33.162326 2576 patch_prober.go:28] interesting pod/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:37038->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 08 03:14:33.162815 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:33.162491 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:37038->10.133.0.39:15021: read: connection reset by peer" Jul 08 03:14:34.053401 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:34.053370 2576 generic.go:356] "Generic (PLEG): container finished" podID="540a6681-39b9-4ed8-998b-d2985cfec180" containerID="23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806" exitCode=0 Jul 08 03:14:34.053551 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:34.053426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" event={"ID":"540a6681-39b9-4ed8-998b-d2985cfec180","Type":"ContainerDied","Data":"23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806"} Jul 08 03:14:34.053765 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:34.053752 2576 scope.go:117] "RemoveContainer" containerID="23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806" Jul 08 03:14:34.054679 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:34.054653 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:14:34.054749 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:34.054741 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:14:34.054794 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:34.054782 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:14:35.060683 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:35.060648 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" event={"ID":"540a6681-39b9-4ed8-998b-d2985cfec180","Type":"ContainerStarted","Data":"86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54"} Jul 08 03:14:35.767245 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:35.767219 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:35.767440 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:35.767345 2576 patch_prober.go:28] interesting pod/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:14:35.767440 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:35.767409 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 08 03:14:36.767371 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:36.767321 2576 patch_prober.go:28] interesting pod/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:14:36.767735 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:36.767400 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 08 03:14:37.766655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:37.766621 2576 patch_prober.go:28] interesting pod/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:14:37.766825 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:37.766670 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 08 03:14:38.273877 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:38.273841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:38.767316 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:38.767286 2576 patch_prober.go:28] interesting pod/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:14:38.767505 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:38.767339 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 08 03:14:39.769986 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:39.769951 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:39.770440 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:39.770223 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:14:39.771003 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:14:39.770988 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:15:00.136482 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.136448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n"] Jul 08 03:15:00.139164 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.139144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:15:00.139284 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.139172 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4"] Jul 08 03:15:00.141722 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.141696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:00.148036 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.147994 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h8t2l\"" Jul 08 03:15:00.152816 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.152794 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n"] Jul 08 03:15:00.155238 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.155214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4"] Jul 08 03:15:00.312812 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.312785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxz2\" (UniqueName: \"kubernetes.io/projected/a1c04e45-69ec-404a-bcb6-fac344eba98d-kube-api-access-pvxz2\") pod \"maas-api-key-cleanup-29724675-4sc8n\" (UID: \"a1c04e45-69ec-404a-bcb6-fac344eba98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:15:00.312946 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.312827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69nm\" (UniqueName: \"kubernetes.io/projected/c3f4d480-01b5-4681-b40b-ef5b171190e0-kube-api-access-f69nm\") pod \"maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4\" (UID: \"c3f4d480-01b5-4681-b40b-ef5b171190e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:00.413275 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.413205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxz2\" (UniqueName: \"kubernetes.io/projected/a1c04e45-69ec-404a-bcb6-fac344eba98d-kube-api-access-pvxz2\") pod \"maas-api-key-cleanup-29724675-4sc8n\" (UID: \"a1c04e45-69ec-404a-bcb6-fac344eba98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:15:00.413275 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.413255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69nm\" (UniqueName: \"kubernetes.io/projected/c3f4d480-01b5-4681-b40b-ef5b171190e0-kube-api-access-f69nm\") pod \"maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4\" (UID: \"c3f4d480-01b5-4681-b40b-ef5b171190e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:00.421274 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.421250 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69nm\" (UniqueName: \"kubernetes.io/projected/c3f4d480-01b5-4681-b40b-ef5b171190e0-kube-api-access-f69nm\") pod \"maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4\" (UID: \"c3f4d480-01b5-4681-b40b-ef5b171190e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:00.421420 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.421403 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxz2\" (UniqueName: \"kubernetes.io/projected/a1c04e45-69ec-404a-bcb6-fac344eba98d-kube-api-access-pvxz2\") pod \"maas-api-key-cleanup-29724675-4sc8n\" (UID: \"a1c04e45-69ec-404a-bcb6-fac344eba98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:15:00.457116 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.457095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:15:00.462925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.462910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:00.599016 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.598979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n"] Jul 08 03:15:00.601529 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:15:00.601498 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c04e45_69ec_404a_bcb6_fac344eba98d.slice/crio-dee0cc31aae1e33e29106a6b5db1e390bbe030a00127c81724b2c9a4af1e1865 WatchSource:0}: Error finding container dee0cc31aae1e33e29106a6b5db1e390bbe030a00127c81724b2c9a4af1e1865: Status 404 returned error can't find the container with id dee0cc31aae1e33e29106a6b5db1e390bbe030a00127c81724b2c9a4af1e1865 Jul 08 03:15:00.603393 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.603379 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:15:00.613679 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:00.613653 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4"] Jul 08 03:15:00.615277 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:15:00.615253 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f4d480_01b5_4681_b40b_ef5b171190e0.slice/crio-87dfc4481d0b0f915163bf6765c949b91daafd4c218253486a171539dfbcf371 WatchSource:0}: Error finding container 87dfc4481d0b0f915163bf6765c949b91daafd4c218253486a171539dfbcf371: Status 404 returned error can't find the container with id 87dfc4481d0b0f915163bf6765c949b91daafd4c218253486a171539dfbcf371 Jul 08 03:15:01.184424 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:01.184393 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" event={"ID":"a1c04e45-69ec-404a-bcb6-fac344eba98d","Type":"ContainerStarted","Data":"dee0cc31aae1e33e29106a6b5db1e390bbe030a00127c81724b2c9a4af1e1865"} Jul 08 03:15:01.185595 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:01.185574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" event={"ID":"c3f4d480-01b5-4681-b40b-ef5b171190e0","Type":"ContainerStarted","Data":"87dfc4481d0b0f915163bf6765c949b91daafd4c218253486a171539dfbcf371"} Jul 08 03:15:03.200046 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:03.199980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" event={"ID":"a1c04e45-69ec-404a-bcb6-fac344eba98d","Type":"ContainerStarted","Data":"ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33"} Jul 08 03:15:03.214549 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:03.214508 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" podStartSLOduration=0.937022219 podStartE2EDuration="3.214493565s" totalImagesPullingTime="2.277471346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:15:03.213339237 +0000 UTC m=+2093.491716438" watchObservedRunningTime="2026-07-08 03:15:03.214493565 +0000 UTC m=+2093.492870765" Jul 08 03:15:04.206184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:04.206148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" event={"ID":"c3f4d480-01b5-4681-b40b-ef5b171190e0","Type":"ContainerStarted","Data":"f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8"} Jul 08 03:15:04.221422 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:04.221340 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" podStartSLOduration=1.9505463459999999 podStartE2EDuration="4.221320879s" totalImagesPullingTime="2.270774533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:15:04.220806185 +0000 UTC m=+2094.499183379" watchObservedRunningTime="2026-07-08 03:15:04.221320879 +0000 UTC m=+2094.499698081" Jul 08 03:15:10.299561 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:10.299532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:15:10.307110 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:10.307083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:15:11.162187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:11.162155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:15:11.508669 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:11.508582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:15:11.513201 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:11.513176 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4"] Jul 08 03:15:11.513432 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:11.513410 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" podUID="c3f4d480-01b5-4681-b40b-ef5b171190e0" containerName="cleanup" containerID="cri-o://f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8" gracePeriod=30 Jul 08 03:15:17.271344 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:17.271303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:15:22.304808 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:22.304779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:15:41.683130 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:41.683096 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:41.813197 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:41.813135 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f69nm\" (UniqueName: \"kubernetes.io/projected/c3f4d480-01b5-4681-b40b-ef5b171190e0-kube-api-access-f69nm\") pod \"c3f4d480-01b5-4681-b40b-ef5b171190e0\" (UID: \"c3f4d480-01b5-4681-b40b-ef5b171190e0\") " Jul 08 03:15:41.815294 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:41.815269 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f4d480-01b5-4681-b40b-ef5b171190e0-kube-api-access-f69nm" (OuterVolumeSpecName: "kube-api-access-f69nm") pod "c3f4d480-01b5-4681-b40b-ef5b171190e0" (UID: "c3f4d480-01b5-4681-b40b-ef5b171190e0"). InnerVolumeSpecName "kube-api-access-f69nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:15:41.914005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:41.913985 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69nm\" (UniqueName: \"kubernetes.io/projected/c3f4d480-01b5-4681-b40b-ef5b171190e0-kube-api-access-f69nm\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:15:42.375736 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.375705 2576 generic.go:356] "Generic (PLEG): container finished" podID="c3f4d480-01b5-4681-b40b-ef5b171190e0" containerID="f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8" exitCode=137 Jul 08 03:15:42.375850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.375743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" event={"ID":"c3f4d480-01b5-4681-b40b-ef5b171190e0","Type":"ContainerDied","Data":"f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8"} Jul 08 03:15:42.375850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.375757 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" Jul 08 03:15:42.375850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.375811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4" event={"ID":"c3f4d480-01b5-4681-b40b-ef5b171190e0","Type":"ContainerDied","Data":"87dfc4481d0b0f915163bf6765c949b91daafd4c218253486a171539dfbcf371"} Jul 08 03:15:42.375850 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.375834 2576 scope.go:117] "RemoveContainer" containerID="f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8" Jul 08 03:15:42.384123 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.384110 2576 scope.go:117] "RemoveContainer" containerID="f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8" Jul 08 03:15:42.384411 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:15:42.384392 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8\": container with ID starting with f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8 not found: ID does not exist" containerID="f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8" Jul 08 03:15:42.384483 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.384418 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8"} err="failed to get container status \"f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8\": rpc error: code = NotFound desc = could not find container \"f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8\": container with ID starting with f39bb81961d2141d9243d0c34d3b9d815a17f9098a09d2407c28d7a9c9aa34b8 not found: ID does not exist" Jul 08 03:15:42.393884 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.393862 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4"] Jul 08 03:15:42.399688 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:42.399668 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c99a305e-29724675-smld4"] Jul 08 03:15:44.278285 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:44.278258 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f4d480-01b5-4681-b40b-ef5b171190e0" path="/var/lib/kubelet/pods/c3f4d480-01b5-4681-b40b-ef5b171190e0/volumes" Jul 08 03:15:57.892931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:57.892899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:15:58.868595 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:58.868560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:15:59.272512 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:15:59.272485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:00.009975 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:00.009949 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n"] Jul 08 03:16:00.010160 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:00.010140 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" podUID="a1c04e45-69ec-404a-bcb6-fac344eba98d" containerName="cleanup" containerID="cri-o://ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33" gracePeriod=30 Jul 08 03:16:04.696934 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:04.696900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:10.402145 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:10.402116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:10.507575 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:10.507515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:10.666542 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:10.666509 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g"] Jul 08 03:16:10.666778 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:10.666749 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" containerID="cri-o://86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54" gracePeriod=30 Jul 08 03:16:10.686314 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:10.686289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:11.972572 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:11.972511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:12.070255 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:12.070225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:12.170490 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:12.170469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:12.899142 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:12.899110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:12.998489 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:12.998457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:13.097495 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:13.097466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:15.917655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:15.917627 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:16:16.078221 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078152 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-istio-token\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078221 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078194 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540a6681-39b9-4ed8-998b-d2985cfec180-istiod-ca-cert\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-certs\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-socket\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078327 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-credential-socket\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078347 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-envoy\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078393 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhtgn\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-kube-api-access-xhtgn\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078456 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078438 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540a6681-39b9-4ed8-998b-d2985cfec180-istio-podinfo\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-data\") pod \"540a6681-39b9-4ed8-998b-d2985cfec180\" (UID: \"540a6681-39b9-4ed8-998b-d2985cfec180\") " Jul 08 03:16:16.078746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078583 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:16:16.078746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078657 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:16:16.078746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078667 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:16:16.078746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078716 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540a6681-39b9-4ed8-998b-d2985cfec180-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:16:16.079005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078859 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.079005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078878 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.079005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078893 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540a6681-39b9-4ed8-998b-d2985cfec180-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.079005 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.078905 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.079382 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.079332 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-data" (OuterVolumeSpecName: "istio-data") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:16:16.080630 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.080607 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:16:16.080703 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.080608 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-kube-api-access-xhtgn" (OuterVolumeSpecName: "kube-api-access-xhtgn") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "kube-api-access-xhtgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:16:16.080882 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.080869 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-istio-token" (OuterVolumeSpecName: "istio-token") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:16:16.080980 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.080962 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/540a6681-39b9-4ed8-998b-d2985cfec180-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "540a6681-39b9-4ed8-998b-d2985cfec180" (UID: "540a6681-39b9-4ed8-998b-d2985cfec180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:16:16.179645 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.179621 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.179645 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.179642 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtgn\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-kube-api-access-xhtgn\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.179770 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.179653 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540a6681-39b9-4ed8-998b-d2985cfec180-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.179770 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.179661 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540a6681-39b9-4ed8-998b-d2985cfec180-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.179770 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.179668 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540a6681-39b9-4ed8-998b-d2985cfec180-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:16.540596 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.540573 2576 generic.go:356] "Generic (PLEG): container finished" podID="540a6681-39b9-4ed8-998b-d2985cfec180" containerID="86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54" exitCode=0 Jul 08 03:16:16.540695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.540627 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" Jul 08 03:16:16.540695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.540658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" event={"ID":"540a6681-39b9-4ed8-998b-d2985cfec180","Type":"ContainerDied","Data":"86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54"} Jul 08 03:16:16.540772 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.540713 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g" event={"ID":"540a6681-39b9-4ed8-998b-d2985cfec180","Type":"ContainerDied","Data":"0a76a4c3d7fc51f574decaedb0098dfe7b32a8c4d4540e11961c956268c9b89c"} Jul 08 03:16:16.540772 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.540737 2576 scope.go:117] "RemoveContainer" containerID="86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54" Jul 08 03:16:16.549150 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.549129 2576 scope.go:117] "RemoveContainer" containerID="23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806" Jul 08 03:16:16.557116 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.557050 2576 scope.go:117] "RemoveContainer" containerID="86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54" Jul 08 03:16:16.557771 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:16:16.557750 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54\": container with ID starting with 86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54 not found: ID does not exist" containerID="86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54" Jul 08 03:16:16.557883 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.557782 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54"} err="failed to get container status \"86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54\": rpc error: code = NotFound desc = could not find container \"86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54\": container with ID starting with 86d92513c45f1e3f655c42b15f74be886311a0c7eaabbb9b97558464a31adb54 not found: ID does not exist" Jul 08 03:16:16.557883 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.557802 2576 scope.go:117] "RemoveContainer" containerID="23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806" Jul 08 03:16:16.558062 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:16:16.558044 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806\": container with ID starting with 23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806 not found: ID does not exist" containerID="23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806" Jul 08 03:16:16.558118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.558072 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806"} err="failed to get container status \"23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806\": rpc error: code = NotFound desc = could not find container \"23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806\": container with ID starting with 23274d4be8c16b92a6fc0da5d5748a01804ab1b01d78d8e3c81753b72e6bc806 not found: ID does not exist" Jul 08 03:16:16.558957 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.558939 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g"] Jul 08 03:16:16.562516 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:16.562497 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c99a305e-openshift-default-57b765cc79-gff7g"] Jul 08 03:16:18.277969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:18.277940 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" path="/var/lib/kubelet/pods/540a6681-39b9-4ed8-998b-d2985cfec180/volumes" Jul 08 03:16:30.194609 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.194587 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:16:30.286328 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.286264 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvxz2\" (UniqueName: \"kubernetes.io/projected/a1c04e45-69ec-404a-bcb6-fac344eba98d-kube-api-access-pvxz2\") pod \"a1c04e45-69ec-404a-bcb6-fac344eba98d\" (UID: \"a1c04e45-69ec-404a-bcb6-fac344eba98d\") " Jul 08 03:16:30.288299 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.288268 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c04e45-69ec-404a-bcb6-fac344eba98d-kube-api-access-pvxz2" (OuterVolumeSpecName: "kube-api-access-pvxz2") pod "a1c04e45-69ec-404a-bcb6-fac344eba98d" (UID: "a1c04e45-69ec-404a-bcb6-fac344eba98d"). InnerVolumeSpecName "kube-api-access-pvxz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:16:30.386784 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.386762 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxz2\" (UniqueName: \"kubernetes.io/projected/a1c04e45-69ec-404a-bcb6-fac344eba98d-kube-api-access-pvxz2\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:16:30.603991 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.603925 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c04e45-69ec-404a-bcb6-fac344eba98d" containerID="ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33" exitCode=137 Jul 08 03:16:30.603991 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.603964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" event={"ID":"a1c04e45-69ec-404a-bcb6-fac344eba98d","Type":"ContainerDied","Data":"ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33"} Jul 08 03:16:30.603991 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.603983 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" Jul 08 03:16:30.604239 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.604002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n" event={"ID":"a1c04e45-69ec-404a-bcb6-fac344eba98d","Type":"ContainerDied","Data":"dee0cc31aae1e33e29106a6b5db1e390bbe030a00127c81724b2c9a4af1e1865"} Jul 08 03:16:30.604239 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.604022 2576 scope.go:117] "RemoveContainer" containerID="ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33" Jul 08 03:16:30.612767 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.612751 2576 scope.go:117] "RemoveContainer" containerID="ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33" Jul 08 03:16:30.612993 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:16:30.612975 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33\": container with ID starting with ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33 not found: ID does not exist" containerID="ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33" Jul 08 03:16:30.613040 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.613000 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33"} err="failed to get container status \"ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33\": rpc error: code = NotFound desc = could not find container \"ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33\": container with ID starting with ef10a25378ea0ccd8697e99bc7efc97873cae8b8c8012a1de484d9ecf972dd33 not found: ID does not exist" Jul 08 03:16:30.624766 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.624741 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n"] Jul 08 03:16:30.628931 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:30.628907 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-4sc8n"] Jul 08 03:16:32.278236 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:32.278207 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c04e45-69ec-404a-bcb6-fac344eba98d" path="/var/lib/kubelet/pods/a1c04e45-69ec-404a-bcb6-fac344eba98d/volumes" Jul 08 03:16:44.365738 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:44.365703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:50.396006 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:50.395970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:55.914206 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:55.914175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:55.975966 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:16:55.975936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:03.976892 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:03.976860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:26.526399 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:26.526340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:26.760118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:26.760088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:26.860571 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:26.860504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:32.391187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:32.391141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:37.848482 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:37.848452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:37.916039 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:37.916009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:38.011790 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:38.011766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:38.105717 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:38.105654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:48.611210 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:48.611169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:17:48.962853 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:17:48.962827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:09.930855 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:09.930819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:15.673091 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:15.673056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:15.773396 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:15.773371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:16.017979 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.017915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:16.297245 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297172 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j"] Jul 08 03:18:16.297539 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297526 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" Jul 08 03:18:16.297591 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297542 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" Jul 08 03:18:16.297591 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297551 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f4d480-01b5-4681-b40b-ef5b171190e0" containerName="cleanup" Jul 08 03:18:16.297591 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297557 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f4d480-01b5-4681-b40b-ef5b171190e0" containerName="cleanup" Jul 08 03:18:16.297591 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297577 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c04e45-69ec-404a-bcb6-fac344eba98d" containerName="cleanup" Jul 08 03:18:16.297591 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297582 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c04e45-69ec-404a-bcb6-fac344eba98d" containerName="cleanup" Jul 08 03:18:16.297738 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297630 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" Jul 08 03:18:16.297738 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297637 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f4d480-01b5-4681-b40b-ef5b171190e0" containerName="cleanup" Jul 08 03:18:16.297738 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.297644 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c04e45-69ec-404a-bcb6-fac344eba98d" containerName="cleanup" Jul 08 03:18:16.299501 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.299486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.301960 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.301938 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-91d71d-openshift-default-dockercfg-jfzgr\"" Jul 08 03:18:16.324374 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.324325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j"] Jul 08 03:18:16.333760 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.333723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:16.344045 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-data\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344138 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-podinfo\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344138 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-certs\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344138 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-token\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344283 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-credential-socket\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344283 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-envoy\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344283 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istiod-ca-cert\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344283 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8xz\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-kube-api-access-vp8xz\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.344283 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.344280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-socket\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.389635 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.389585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:16.444841 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.444803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-data\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.444841 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.444838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-podinfo\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445062 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.444861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-certs\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445062 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.444981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-token\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445062 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-credential-socket\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445216 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-envoy\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445346 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-certs\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445346 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-data\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445522 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-credential-socket\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445522 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istiod-ca-cert\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445522 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8xz\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-kube-api-access-vp8xz\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445668 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-socket\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445848 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-socket\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.445986 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.445964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istiod-ca-cert\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.447840 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.447822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-envoy\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.447965 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.447949 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-podinfo\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.453618 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.453592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-token\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.453894 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.453876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8xz\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-kube-api-access-vp8xz\") pod \"e2e-inf-a-91d71d-openshift-default-85bc987665-7796j\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.610605 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.610549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:16.748238 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.748184 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j"] Jul 08 03:18:16.751499 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:18:16.751472 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4e17d6_8090_453b_8b1c_03a1f7ac7eaf.slice/crio-1dc84045b0f7f0e8d49133cbf53ee1ef190087dcd8b22d6539e5ef8715943b72 WatchSource:0}: Error finding container 1dc84045b0f7f0e8d49133cbf53ee1ef190087dcd8b22d6539e5ef8715943b72: Status 404 returned error can't find the container with id 1dc84045b0f7f0e8d49133cbf53ee1ef190087dcd8b22d6539e5ef8715943b72 Jul 08 03:18:16.754069 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.754042 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:16.754154 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.754104 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:16.754154 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:16.754132 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:17.070679 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:17.070642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" event={"ID":"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf","Type":"ContainerStarted","Data":"514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60"} Jul 08 03:18:17.070865 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:17.070695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" event={"ID":"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf","Type":"ContainerStarted","Data":"1dc84045b0f7f0e8d49133cbf53ee1ef190087dcd8b22d6539e5ef8715943b72"} Jul 08 03:18:17.090461 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:17.090414 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podStartSLOduration=1.090382838 podStartE2EDuration="1.090382838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:18:17.088730042 +0000 UTC m=+2287.367107243" watchObservedRunningTime="2026-07-08 03:18:17.090382838 +0000 UTC m=+2287.368760038" Jul 08 03:18:17.611146 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:17.611121 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:17.615557 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:17.615534 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:18.075506 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:18.075482 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:18.076573 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:18.076551 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:19.009023 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:19.008980 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:19.099252 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:19.099220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:19.163918 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:19.163889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:19.470038 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:19.470011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:19.743061 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:19.742986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:19.846566 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:19.846544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:23.507449 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:23.507418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:23.850164 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:23.850083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:24.118483 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.118407 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb"] Jul 08 03:18:24.118814 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.118797 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" Jul 08 03:18:24.118898 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.118816 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" Jul 08 03:18:24.118953 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.118931 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="540a6681-39b9-4ed8-998b-d2985cfec180" containerName="istio-proxy" Jul 08 03:18:24.121121 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.121101 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.123412 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.123390 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a6e00a-openshift-default-dockercfg-957z5\"" Jul 08 03:18:24.131640 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.131610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb"] Jul 08 03:18:24.153313 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.153282 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:24.209613 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.209586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istiod-ca-cert\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.209801 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.209771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-token\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.209974 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.209951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-podinfo\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.210110 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.210084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-credential-socket\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.210229 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.210150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-certs\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.210229 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.210185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvnp\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-kube-api-access-xdvnp\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.210229 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.210222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-envoy\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.210424 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.210298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-data\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.210424 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.210341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-socket\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.286753 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.286723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:24.311023 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.310993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-certs\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvnp\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-kube-api-access-xdvnp\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-envoy\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-data\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-socket\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istiod-ca-cert\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311464 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-token\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311464 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-podinfo\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311573 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311470 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-certs\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311573 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-data\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311573 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-credential-socket\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311729 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-socket\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-credential-socket\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.311914 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.311895 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istiod-ca-cert\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.313550 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.313523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-envoy\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.314001 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.313980 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-podinfo\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.318423 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.318400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-token\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.318580 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.318561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvnp\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-kube-api-access-xdvnp\") pod \"e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.432979 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.432952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:24.572552 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.572528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb"] Jul 08 03:18:24.574410 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:18:24.574385 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec7fe3d_6ca8_4448_9767_38c4027343d4.slice/crio-b9796cad3b19e441ab6d57e51f52079a79638f51cc4752662bb89fc22074bc1b WatchSource:0}: Error finding container b9796cad3b19e441ab6d57e51f52079a79638f51cc4752662bb89fc22074bc1b: Status 404 returned error can't find the container with id b9796cad3b19e441ab6d57e51f52079a79638f51cc4752662bb89fc22074bc1b Jul 08 03:18:24.576820 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.576789 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:24.576885 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.576853 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:24.576885 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:24.576881 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:25.110567 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:25.110525 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" event={"ID":"8ec7fe3d-6ca8-4448-9767-38c4027343d4","Type":"ContainerStarted","Data":"12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3"} Jul 08 03:18:25.110761 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:25.110590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" event={"ID":"8ec7fe3d-6ca8-4448-9767-38c4027343d4","Type":"ContainerStarted","Data":"b9796cad3b19e441ab6d57e51f52079a79638f51cc4752662bb89fc22074bc1b"} Jul 08 03:18:25.133465 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:25.133422 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podStartSLOduration=1.1334048939999999 podStartE2EDuration="1.133404894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:18:25.132564709 +0000 UTC m=+2295.410941936" watchObservedRunningTime="2026-07-08 03:18:25.133404894 +0000 UTC m=+2295.411782097" Jul 08 03:18:25.433525 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:25.433495 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:25.438054 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:25.438032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:26.114801 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:26.114771 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:26.115753 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:26.115733 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:27.027692 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:27.027664 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:27.104730 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:27.104679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:27.167846 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:27.167805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:27.370555 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:27.370474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:27.781262 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:27.781223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:27.879109 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:27.879078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:31.559173 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:31.559139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:34.117528 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:34.117491 2576 patch_prober.go:28] interesting pod/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 03:18:34.117977 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:34.117585 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 03:18:35.159941 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:35.159904 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerID="514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60" exitCode=0 Jul 08 03:18:35.160433 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:35.159981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" event={"ID":"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf","Type":"ContainerDied","Data":"514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60"} Jul 08 03:18:35.160433 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:35.160423 2576 scope.go:117] "RemoveContainer" containerID="514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60" Jul 08 03:18:35.161462 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:35.161427 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:35.161577 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:35.161501 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:35.161577 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:35.161535 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:36.166676 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:36.166644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" event={"ID":"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf","Type":"ContainerStarted","Data":"71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c"} Jul 08 03:18:36.612067 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:36.611970 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:36.612517 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:36.612424 2576 patch_prober.go:28] interesting pod/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 08 03:18:36.612710 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:36.612685 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 08 03:18:37.611683 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:37.611638 2576 patch_prober.go:28] interesting pod/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 08 03:18:37.612185 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:37.611730 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 08 03:18:38.064586 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:38.064556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:38.611978 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:38.611938 2576 patch_prober.go:28] interesting pod/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 08 03:18:38.612408 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:38.612006 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 08 03:18:39.612190 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:39.611731 2576 patch_prober.go:28] interesting pod/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 08 03:18:39.612190 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:39.611792 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 08 03:18:40.495605 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:40.495549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:40.595260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:40.595220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:40.615725 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:40.615702 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:40.616240 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:40.616206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:40.616810 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:40.616790 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:18:41.700053 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:41.699643 2576 patch_prober.go:28] interesting pod/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:38982->10.133.0.43:15021: read: connection reset by peer" start-of-body= Jul 08 03:18:41.700053 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:41.699717 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:38982->10.133.0.43:15021: read: connection reset by peer" Jul 08 03:18:42.195445 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:42.195417 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerID="12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3" exitCode=0 Jul 08 03:18:42.195587 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:42.195494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" event={"ID":"8ec7fe3d-6ca8-4448-9767-38c4027343d4","Type":"ContainerDied","Data":"12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3"} Jul 08 03:18:42.196040 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:42.196024 2576 scope.go:117] "RemoveContainer" containerID="12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3" Jul 08 03:18:42.196989 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:42.196962 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:42.197077 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:42.197036 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:42.197123 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:42.197078 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:18:43.206683 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:43.206639 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" event={"ID":"8ec7fe3d-6ca8-4448-9767-38c4027343d4","Type":"ContainerStarted","Data":"0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7"} Jul 08 03:18:43.433470 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:43.433435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:43.433685 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:43.433663 2576 patch_prober.go:28] interesting pod/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 08 03:18:43.433741 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:43.433710 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 08 03:18:44.433935 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:44.433902 2576 patch_prober.go:28] interesting pod/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 08 03:18:44.434304 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:44.433988 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 08 03:18:45.434298 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:45.434258 2576 patch_prober.go:28] interesting pod/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 08 03:18:45.434700 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:45.434327 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 08 03:18:46.434138 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:46.434103 2576 patch_prober.go:28] interesting pod/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 08 03:18:46.434318 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:46.434170 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 08 03:18:47.436976 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:47.436940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:47.437438 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:47.437259 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:47.437879 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:47.437866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:18:48.765461 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:48.765430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:49.987452 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:49.987399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:51.583820 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:51.583775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:51.684793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:18:51.684769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:14.756842 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:14.756760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:22.088612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:22.088580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:34.680760 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:34.680727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:34.956831 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:34.956739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn"] Jul 08 03:19:34.959154 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:34.959137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:34.962954 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:34.962639 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-57a7d5-openshift-default-dockercfg-xqswn\"" Jul 08 03:19:34.983449 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:34.983428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn"] Jul 08 03:19:35.019944 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.019914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:35.087520 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-credential-socket\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087723 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea24240-ad46-4350-a59c-1bd70311c526-istio-podinfo\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087723 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-socket\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087723 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-envoy\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087893 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-certs\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087893 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea24240-ad46-4350-a59c-1bd70311c526-istiod-ca-cert\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087893 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6kl\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-kube-api-access-2b6kl\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.087893 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-data\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.088030 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.087940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-istio-token\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.188663 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-data\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.188800 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-istio-token\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.188800 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-credential-socket\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.188800 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea24240-ad46-4350-a59c-1bd70311c526-istio-podinfo\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.188956 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-socket\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.188956 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-envoy\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189066 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.188978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-certs\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189066 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea24240-ad46-4350-a59c-1bd70311c526-istiod-ca-cert\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6kl\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-kube-api-access-2b6kl\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189184 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189105 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-data\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189567 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189200 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-credential-socket\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189567 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-socket\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189567 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189388 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-certs\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.189700 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.189675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea24240-ad46-4350-a59c-1bd70311c526-istiod-ca-cert\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.191386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.191346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-envoy\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.191608 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.191592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea24240-ad46-4350-a59c-1bd70311c526-istio-podinfo\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.196168 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.196144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-istio-token\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.196584 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.196564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6kl\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-kube-api-access-2b6kl\") pod \"e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.246598 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.246542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:35.275001 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.274979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:35.413806 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.413784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn"] Jul 08 03:19:35.415685 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:19:35.415656 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea24240_ad46_4350_a59c_1bd70311c526.slice/crio-7ad6f943c43fd15d35586d1b4509cd2b0f51076a0dcc215d4e8b50155dca5cb3 WatchSource:0}: Error finding container 7ad6f943c43fd15d35586d1b4509cd2b0f51076a0dcc215d4e8b50155dca5cb3: Status 404 returned error can't find the container with id 7ad6f943c43fd15d35586d1b4509cd2b0f51076a0dcc215d4e8b50155dca5cb3 Jul 08 03:19:35.417844 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.417810 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:19:35.417938 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.417875 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:19:35.417938 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.417905 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:19:35.457343 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:35.457308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" event={"ID":"9ea24240-ad46-4350-a59c-1bd70311c526","Type":"ContainerStarted","Data":"7ad6f943c43fd15d35586d1b4509cd2b0f51076a0dcc215d4e8b50155dca5cb3"} Jul 08 03:19:36.463023 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:36.462988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" event={"ID":"9ea24240-ad46-4350-a59c-1bd70311c526","Type":"ContainerStarted","Data":"1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa"} Jul 08 03:19:36.486036 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:36.485992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podStartSLOduration=2.485977542 podStartE2EDuration="2.485977542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:19:36.483486093 +0000 UTC m=+2366.761863331" watchObservedRunningTime="2026-07-08 03:19:36.485977542 +0000 UTC m=+2366.764354743" Jul 08 03:19:37.275694 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:37.275662 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:37.280729 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:37.280704 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:37.467653 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:37.467607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:37.468548 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:37.468529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:38.381276 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:38.381241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:38.471678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:38.471648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:38.580298 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:38.580265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:38.772176 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:38.772095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:39.030294 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:39.030221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:39.130520 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:39.130486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:42.878415 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:42.878388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:19:51.544608 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:51.544577 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ea24240-ad46-4350-a59c-1bd70311c526" containerID="1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa" exitCode=0 Jul 08 03:19:51.545020 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:51.544625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" event={"ID":"9ea24240-ad46-4350-a59c-1bd70311c526","Type":"ContainerDied","Data":"1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa"} Jul 08 03:19:51.545020 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:51.544919 2576 scope.go:117] "RemoveContainer" containerID="1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa" Jul 08 03:19:51.545908 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:51.545874 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:19:51.546004 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:51.545957 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:19:51.546065 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:51.546015 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:19:52.468046 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:52.468011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:52.551640 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:52.551607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" event={"ID":"9ea24240-ad46-4350-a59c-1bd70311c526","Type":"ContainerStarted","Data":"11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4"} Jul 08 03:19:53.275677 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:53.275646 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:53.276423 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:53.276402 2576 patch_prober.go:28] interesting pod/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 03:19:53.276548 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:53.276446 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 03:19:54.275333 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:54.275302 2576 patch_prober.go:28] interesting pod/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 03:19:54.275756 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:54.275370 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 03:19:55.276274 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:55.276240 2576 patch_prober.go:28] interesting pod/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 03:19:55.276682 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:55.276304 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 03:19:56.275464 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:56.275432 2576 patch_prober.go:28] interesting pod/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 03:19:56.275634 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:56.275480 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 03:19:57.279261 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:57.279234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:57.279702 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:57.279553 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:19:57.280271 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:19:57.280252 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:20:04.273328 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:04.273256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:20:10.328839 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:10.328809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:20:10.338029 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:10.338009 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:20:10.377601 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:10.377581 2576 scope.go:117] "RemoveContainer" containerID="29d40630a37adbe10ec2c70f9261b2bd01581df4016c7b2bf06f7ca907cb8a6c" Jul 08 03:20:19.969151 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:19.969085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:20:20.358467 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:20.358397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:20:20.434929 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:20.434899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:20:25.374402 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:20:25.374340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:02.094724 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:02.094662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:02.515315 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:02.515249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:08.007273 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:08.007230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:08.107621 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:08.107567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:08.269911 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:08.269841 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn"] Jul 08 03:21:08.270098 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:08.270077 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" containerID="cri-o://11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4" gracePeriod=30 Jul 08 03:21:08.281117 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:08.281088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:09.457834 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:09.457803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:10.464334 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:10.464308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:10.763220 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:10.763126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:11.122779 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:11.122701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:11.250194 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:11.250166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:11.362866 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:11.362841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:11.520831 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:11.520797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:12.281872 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:12.281845 2576 patch_prober.go:28] interesting pod/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 03:21:12.282028 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:12.281905 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 03:21:13.523400 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.523376 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:21:13.679748 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-socket\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.679891 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-credential-socket\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.679891 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679775 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-istio-token\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.679891 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679803 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea24240-ad46-4350-a59c-1bd70311c526-istiod-ca-cert\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.679891 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679844 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-envoy\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.679891 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679884 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b6kl\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-kube-api-access-2b6kl\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.680131 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679959 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-data\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.680131 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679984 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-certs\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.680131 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.679981 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:13.680131 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680010 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea24240-ad46-4350-a59c-1bd70311c526-istio-podinfo\") pod \"9ea24240-ad46-4350-a59c-1bd70311c526\" (UID: \"9ea24240-ad46-4350-a59c-1bd70311c526\") " Jul 08 03:21:13.680131 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680061 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:13.680476 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680249 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:13.680476 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680443 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.680476 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680462 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.680819 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680476 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.680819 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680520 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea24240-ad46-4350-a59c-1bd70311c526-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:21:13.681013 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.680984 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:13.682265 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.682237 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-kube-api-access-2b6kl" (OuterVolumeSpecName: "kube-api-access-2b6kl") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "kube-api-access-2b6kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:21:13.682371 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.682260 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:21:13.682371 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.682270 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ea24240-ad46-4350-a59c-1bd70311c526-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:21:13.682455 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.682399 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ea24240-ad46-4350-a59c-1bd70311c526" (UID: "9ea24240-ad46-4350-a59c-1bd70311c526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:13.780994 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.780971 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.780994 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.780993 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b6kl\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-kube-api-access-2b6kl\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.781118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.781003 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea24240-ad46-4350-a59c-1bd70311c526-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.781118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.781011 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea24240-ad46-4350-a59c-1bd70311c526-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.781118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.781019 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea24240-ad46-4350-a59c-1bd70311c526-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.781118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.781027 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea24240-ad46-4350-a59c-1bd70311c526-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:13.927808 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.927785 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ea24240-ad46-4350-a59c-1bd70311c526" containerID="11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4" exitCode=0 Jul 08 03:21:13.927909 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.927833 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" Jul 08 03:21:13.927909 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.927858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" event={"ID":"9ea24240-ad46-4350-a59c-1bd70311c526","Type":"ContainerDied","Data":"11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4"} Jul 08 03:21:13.927976 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.927918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn" event={"ID":"9ea24240-ad46-4350-a59c-1bd70311c526","Type":"ContainerDied","Data":"7ad6f943c43fd15d35586d1b4509cd2b0f51076a0dcc215d4e8b50155dca5cb3"} Jul 08 03:21:13.927976 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.927939 2576 scope.go:117] "RemoveContainer" containerID="11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4" Jul 08 03:21:13.936981 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.936966 2576 scope.go:117] "RemoveContainer" containerID="1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa" Jul 08 03:21:13.944766 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.944752 2576 scope.go:117] "RemoveContainer" containerID="11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4" Jul 08 03:21:13.945003 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:21:13.944975 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4\": container with ID starting with 11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4 not found: ID does not exist" containerID="11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4" Jul 08 03:21:13.945051 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.945002 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4"} err="failed to get container status \"11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4\": rpc error: code = NotFound desc = could not find container \"11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4\": container with ID starting with 11e56b83d3781da404792b479b4e4bbbc065092aa758764ee91506ddde872ae4 not found: ID does not exist" Jul 08 03:21:13.945051 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.945019 2576 scope.go:117] "RemoveContainer" containerID="1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa" Jul 08 03:21:13.945261 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:21:13.945244 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa\": container with ID starting with 1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa not found: ID does not exist" containerID="1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa" Jul 08 03:21:13.945301 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.945268 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa"} err="failed to get container status \"1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa\": rpc error: code = NotFound desc = could not find container \"1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa\": container with ID starting with 1917f5cb9080316ac2a3de314234fed105fdbf23807ba162f8b38771e24022fa not found: ID does not exist" Jul 08 03:21:13.950288 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.950271 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn"] Jul 08 03:21:13.953716 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:13.953696 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-reuse-57a7d5-openshift-default-85c5446df-m9jmn"] Jul 08 03:21:14.278732 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:14.278675 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" path="/var/lib/kubelet/pods/9ea24240-ad46-4350-a59c-1bd70311c526/volumes" Jul 08 03:21:16.197871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:16.197842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:53.146485 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:53.146442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:53.406111 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:53.406026 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j"] Jul 08 03:21:53.406285 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:53.406247 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" containerID="cri-o://71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c" gracePeriod=30 Jul 08 03:21:53.419718 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:53.419691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:53.814795 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:53.814723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:54.221744 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:54.221709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:54.322121 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:54.322087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:54.422696 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:54.422661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:54.522404 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:54.522304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:55.617950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:55.617920 2576 patch_prober.go:28] interesting pod/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 03:21:55.618305 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:55.617980 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 03:21:58.653929 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.653906 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:21:58.720743 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720687 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-credential-socket\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.720743 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720729 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-certs\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.720925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720780 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp8xz\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-kube-api-access-vp8xz\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.720925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720820 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-token\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.720925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720849 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-envoy\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.720925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720901 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-socket\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.721134 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720959 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-data\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.721134 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720977 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:58.721134 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.720988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istiod-ca-cert\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.721134 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721049 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-podinfo\") pod \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\" (UID: \"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf\") " Jul 08 03:21:58.721134 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721091 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:58.721427 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721280 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:58.721427 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721409 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.721542 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721453 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.721542 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721469 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.721631 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.721571 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:21:58.722243 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.722218 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:58.723268 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.723233 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:21:58.723394 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.723313 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:21:58.723506 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.723489 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-kube-api-access-vp8xz" (OuterVolumeSpecName: "kube-api-access-vp8xz") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "kube-api-access-vp8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:21:58.723506 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.723488 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" (UID: "7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:21:58.822191 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.822169 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.822191 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.822190 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.822322 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.822199 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.822322 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.822206 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8xz\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-kube-api-access-vp8xz\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.822322 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.822215 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:58.822322 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:58.822222 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:21:59.126051 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.126022 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerID="71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c" exitCode=0 Jul 08 03:21:59.126164 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.126072 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" Jul 08 03:21:59.126164 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.126103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" event={"ID":"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf","Type":"ContainerDied","Data":"71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c"} Jul 08 03:21:59.126164 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.126153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j" event={"ID":"7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf","Type":"ContainerDied","Data":"1dc84045b0f7f0e8d49133cbf53ee1ef190087dcd8b22d6539e5ef8715943b72"} Jul 08 03:21:59.126267 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.126174 2576 scope.go:117] "RemoveContainer" containerID="71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c" Jul 08 03:21:59.135524 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.135497 2576 scope.go:117] "RemoveContainer" containerID="514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60" Jul 08 03:21:59.143733 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.143711 2576 scope.go:117] "RemoveContainer" containerID="71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c" Jul 08 03:21:59.143960 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:21:59.143944 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c\": container with ID starting with 71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c not found: ID does not exist" containerID="71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c" Jul 08 03:21:59.144016 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.143966 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c"} err="failed to get container status \"71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c\": rpc error: code = NotFound desc = could not find container \"71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c\": container with ID starting with 71dd2179d4142e7093d4242371dd70829a1aa97bb5560e1e9a4570ccbe92f36c not found: ID does not exist" Jul 08 03:21:59.144016 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.143982 2576 scope.go:117] "RemoveContainer" containerID="514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60" Jul 08 03:21:59.144200 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:21:59.144175 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60\": container with ID starting with 514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60 not found: ID does not exist" containerID="514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60" Jul 08 03:21:59.144329 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.144210 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60"} err="failed to get container status \"514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60\": rpc error: code = NotFound desc = could not find container \"514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60\": container with ID starting with 514940b5edbcb3c9ce343a3281fef12a309b12019f3f39751db24e72d004db60 not found: ID does not exist" Jul 08 03:21:59.147277 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.147253 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j"] Jul 08 03:21:59.153697 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:21:59.153678 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-91d71d-openshift-default-85bc987665-7796j"] Jul 08 03:22:00.001961 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.001934 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:00.115145 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.115121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:00.291916 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.291827 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" path="/var/lib/kubelet/pods/7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf/volumes" Jul 08 03:22:00.292170 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.292151 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb"] Jul 08 03:22:00.292454 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.292431 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" containerID="cri-o://0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7" gracePeriod=30 Jul 08 03:22:00.319123 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.319098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:00.875668 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:00.875634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:01.170163 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170075 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr"] Jul 08 03:22:01.170621 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170590 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" Jul 08 03:22:01.170621 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170608 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" Jul 08 03:22:01.170746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170636 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" Jul 08 03:22:01.170746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170645 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" Jul 08 03:22:01.170746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" Jul 08 03:22:01.170746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170662 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" Jul 08 03:22:01.170746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170683 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" Jul 08 03:22:01.170746 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170691 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" Jul 08 03:22:01.171016 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170775 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" Jul 08 03:22:01.171016 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170788 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" Jul 08 03:22:01.171016 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.170799 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4e17d6-8090-453b-8b1c-03a1f7ac7eaf" containerName="istio-proxy" Jul 08 03:22:01.174070 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.174049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.176572 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.176550 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ed1785-openshift-default-dockercfg-7fb6d\"" Jul 08 03:22:01.186443 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.186423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr"] Jul 08 03:22:01.202792 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.202768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:01.240963 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.240937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe66094-f26f-48f8-acc6-378766fa7854-istiod-ca-cert\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241088 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.240973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe66094-f26f-48f8-acc6-378766fa7854-istio-podinfo\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241088 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-credential-socket\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241088 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-istio-token\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241247 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-certs\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241247 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-envoy\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241247 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-data\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241247 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-socket\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.241247 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.241225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmx7\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-kube-api-access-9jmx7\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342179 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-envoy\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342179 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-data\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-socket\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmx7\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-kube-api-access-9jmx7\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe66094-f26f-48f8-acc6-378766fa7854-istiod-ca-cert\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe66094-f26f-48f8-acc6-378766fa7854-istio-podinfo\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-credential-socket\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-istio-token\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342749 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-socket\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342749 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-certs\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.342749 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.342730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-data\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.343071 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.343048 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe66094-f26f-48f8-acc6-378766fa7854-istiod-ca-cert\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.343268 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.343251 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-certs\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.343477 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.343456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-credential-socket\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.344694 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.344677 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-envoy\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.344927 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.344909 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe66094-f26f-48f8-acc6-378766fa7854-istio-podinfo\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.351614 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.351585 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-istio-token\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.351925 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.351906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmx7\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-kube-api-access-9jmx7\") pod \"e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.465946 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.465878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:01.485975 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.485953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:01.622268 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.622243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr"] Jul 08 03:22:01.625185 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:22:01.625155 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe66094_f26f_48f8_acc6_378766fa7854.slice/crio-e998759ddf9c582bfeefe7bd413e75cbbb7d473f939c0b452d1d7928cd9262d5 WatchSource:0}: Error finding container e998759ddf9c582bfeefe7bd413e75cbbb7d473f939c0b452d1d7928cd9262d5: Status 404 returned error can't find the container with id e998759ddf9c582bfeefe7bd413e75cbbb7d473f939c0b452d1d7928cd9262d5 Jul 08 03:22:01.627699 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.627664 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:01.627788 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.627731 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:01.627788 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:01.627759 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:02.143539 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.143505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" event={"ID":"dfe66094-f26f-48f8-acc6-378766fa7854","Type":"ContainerStarted","Data":"2f8cd28d66d4d58a76eda0e38565b598179b93fe69fd2742ef081d0720eed9f9"} Jul 08 03:22:02.143703 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.143553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" event={"ID":"dfe66094-f26f-48f8-acc6-378766fa7854","Type":"ContainerStarted","Data":"e998759ddf9c582bfeefe7bd413e75cbbb7d473f939c0b452d1d7928cd9262d5"} Jul 08 03:22:02.165671 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.165629 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podStartSLOduration=1.165614524 podStartE2EDuration="1.165614524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:22:02.163368393 +0000 UTC m=+2512.441745592" watchObservedRunningTime="2026-07-08 03:22:02.165614524 +0000 UTC m=+2512.443991724" Jul 08 03:22:02.439450 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.439374 2576 patch_prober.go:28] interesting pod/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 03:22:02.439450 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.439422 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 03:22:02.486412 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.486380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:02.490675 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:02.490652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:03.148074 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:03.148043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:03.148923 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:03.148903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:04.070630 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:04.070597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:04.140113 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:04.140076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:04.241296 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:04.241269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:04.341258 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:04.341183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:04.667328 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:04.667293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:04.773652 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:04.773620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:05.542632 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.542611 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:22:05.679132 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679107 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-data\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679137 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-credential-socket\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679169 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istiod-ca-cert\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679189 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdvnp\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-kube-api-access-xdvnp\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-envoy\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679239 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-token\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-podinfo\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679319 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-certs\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679374 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-socket\") pod \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\" (UID: \"8ec7fe3d-6ca8-4448-9767-38c4027343d4\") " Jul 08 03:22:05.679695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679402 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:05.679695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679597 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.679695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679626 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:05.679695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679655 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:22:05.679986 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679796 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:05.679986 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.679890 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:05.681631 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.681608 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:05.681861 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.681840 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-kube-api-access-xdvnp" (OuterVolumeSpecName: "kube-api-access-xdvnp") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "kube-api-access-xdvnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:22:05.682127 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.682098 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:22:05.682262 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.682106 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ec7fe3d-6ca8-4448-9767-38c4027343d4" (UID: "8ec7fe3d-6ca8-4448-9767-38c4027343d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:22:05.779971 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.779949 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.779971 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.779969 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.780101 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.779977 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdvnp\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-kube-api-access-xdvnp\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.780101 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.779985 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.780101 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.779993 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.780101 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.780001 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec7fe3d-6ca8-4448-9767-38c4027343d4-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.780101 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.780009 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:05.780101 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:05.780018 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec7fe3d-6ca8-4448-9767-38c4027343d4-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:06.162822 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.162793 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerID="0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7" exitCode=0 Jul 08 03:22:06.162975 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.162845 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" Jul 08 03:22:06.162975 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.162867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" event={"ID":"8ec7fe3d-6ca8-4448-9767-38c4027343d4","Type":"ContainerDied","Data":"0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7"} Jul 08 03:22:06.162975 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.162907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb" event={"ID":"8ec7fe3d-6ca8-4448-9767-38c4027343d4","Type":"ContainerDied","Data":"b9796cad3b19e441ab6d57e51f52079a79638f51cc4752662bb89fc22074bc1b"} Jul 08 03:22:06.162975 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.162929 2576 scope.go:117] "RemoveContainer" containerID="0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7" Jul 08 03:22:06.172038 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.172021 2576 scope.go:117] "RemoveContainer" containerID="12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3" Jul 08 03:22:06.179626 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.179608 2576 scope.go:117] "RemoveContainer" containerID="0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7" Jul 08 03:22:06.179869 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:22:06.179849 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7\": container with ID starting with 0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7 not found: ID does not exist" containerID="0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7" Jul 08 03:22:06.179911 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.179878 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7"} err="failed to get container status \"0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7\": rpc error: code = NotFound desc = could not find container \"0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7\": container with ID starting with 0767558ec0e01c4ce73ea45b7c643b3c4d2e87c312df5292c91db0a628be61b7 not found: ID does not exist" Jul 08 03:22:06.179911 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.179895 2576 scope.go:117] "RemoveContainer" containerID="12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3" Jul 08 03:22:06.180128 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:22:06.180108 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3\": container with ID starting with 12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3 not found: ID does not exist" containerID="12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3" Jul 08 03:22:06.180246 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.180134 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3"} err="failed to get container status \"12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3\": rpc error: code = NotFound desc = could not find container \"12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3\": container with ID starting with 12c2466b3f9455a43d80bf11c926f9cbb8545d6dfbbf84c02a88e02949f210b3 not found: ID does not exist" Jul 08 03:22:06.186499 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.186475 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb"] Jul 08 03:22:06.193749 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.193730 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a6e00a-openshift-default-79d8fc87b9-tqvhb"] Jul 08 03:22:06.278011 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:06.277987 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" path="/var/lib/kubelet/pods/8ec7fe3d-6ca8-4448-9767-38c4027343d4/volumes" Jul 08 03:22:08.562882 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:08.562854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:09.030454 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.030420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:09.318784 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.318694 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr"] Jul 08 03:22:09.319075 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.319059 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" Jul 08 03:22:09.319075 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.319075 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" Jul 08 03:22:09.319220 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.319088 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" Jul 08 03:22:09.319220 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.319094 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" Jul 08 03:22:09.319220 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.319151 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" Jul 08 03:22:09.319220 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.319163 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea24240-ad46-4350-a59c-1bd70311c526" containerName="istio-proxy" Jul 08 03:22:09.321457 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.321434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.324273 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.324252 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a4e50e-openshift-default-dockercfg-7gnkn\"" Jul 08 03:22:09.333678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.333657 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr"] Jul 08 03:22:09.350756 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.350728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:09.406409 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-data\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406409 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-podinfo\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-certs\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-envoy\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-socket\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406768 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-credential-socket\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406768 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-token\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406768 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba0f02c-bdf2-4e64-b2a9-471f81510401-istiod-ca-cert\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.406922 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.406789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdmr\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-kube-api-access-pwdmr\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507696 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba0f02c-bdf2-4e64-b2a9-471f81510401-istiod-ca-cert\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507696 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdmr\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-kube-api-access-pwdmr\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-data\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-podinfo\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-certs\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-envoy\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-socket\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.507910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.507851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-credential-socket\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.508163 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.508141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-token\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.508297 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.508269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-socket\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.508384 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.508315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-data\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.508443 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.508397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-credential-socket\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.508581 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.508563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-certs\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.508636 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.508603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba0f02c-bdf2-4e64-b2a9-471f81510401-istiod-ca-cert\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.510426 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.510392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-envoy\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.511020 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.510995 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-podinfo\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.516541 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.516520 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-token\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.517200 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.517180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdmr\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-kube-api-access-pwdmr\") pod \"e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.634397 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.634345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:09.639157 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.639133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:09.769531 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.769508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr"] Jul 08 03:22:09.771580 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:22:09.771553 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba0f02c_bdf2_4e64_b2a9_471f81510401.slice/crio-aae511b07e7cfa9575b875a1b451fa6519e8120819eeb1ce7d0a1afbad7443ae WatchSource:0}: Error finding container aae511b07e7cfa9575b875a1b451fa6519e8120819eeb1ce7d0a1afbad7443ae: Status 404 returned error can't find the container with id aae511b07e7cfa9575b875a1b451fa6519e8120819eeb1ce7d0a1afbad7443ae Jul 08 03:22:09.773869 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.773818 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:09.773945 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.773914 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:09.773995 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:09.773958 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:10.184103 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:10.184066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" event={"ID":"bba0f02c-bdf2-4e64-b2a9-471f81510401","Type":"ContainerStarted","Data":"8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96"} Jul 08 03:22:10.184299 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:10.184120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" event={"ID":"bba0f02c-bdf2-4e64-b2a9-471f81510401","Type":"ContainerStarted","Data":"aae511b07e7cfa9575b875a1b451fa6519e8120819eeb1ce7d0a1afbad7443ae"} Jul 08 03:22:10.205000 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:10.204953 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podStartSLOduration=1.204936133 podStartE2EDuration="1.204936133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:22:10.202806146 +0000 UTC m=+2520.481183357" watchObservedRunningTime="2026-07-08 03:22:10.204936133 +0000 UTC m=+2520.483313336" Jul 08 03:22:10.635073 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:10.635049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:10.639900 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:10.639880 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:11.189383 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:11.189320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:11.190133 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:11.190116 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:12.098770 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:12.098739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:12.190317 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:12.190283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:12.252935 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:12.252903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:12.353126 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:12.353026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:12.757115 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:12.757076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:12.860909 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:12.860879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:16.615921 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:16.615883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:19.189498 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:19.189460 2576 patch_prober.go:28] interesting pod/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 03:22:19.189871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:19.189535 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 03:22:21.246662 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:21.246627 2576 generic.go:356] "Generic (PLEG): container finished" podID="dfe66094-f26f-48f8-acc6-378766fa7854" containerID="2f8cd28d66d4d58a76eda0e38565b598179b93fe69fd2742ef081d0720eed9f9" exitCode=0 Jul 08 03:22:21.248437 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:21.247272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" event={"ID":"dfe66094-f26f-48f8-acc6-378766fa7854","Type":"ContainerDied","Data":"2f8cd28d66d4d58a76eda0e38565b598179b93fe69fd2742ef081d0720eed9f9"} Jul 08 03:22:21.248437 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:21.247824 2576 scope.go:117] "RemoveContainer" containerID="2f8cd28d66d4d58a76eda0e38565b598179b93fe69fd2742ef081d0720eed9f9" Jul 08 03:22:21.250203 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:21.250155 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:21.250462 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:21.250422 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:21.251221 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:21.251114 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:22.253500 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:22.253467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" event={"ID":"dfe66094-f26f-48f8-acc6-378766fa7854","Type":"ContainerStarted","Data":"ed9bac5e9dced14814d4e0066ccd36c9513db40f458fa55db991309d1cf660c0"} Jul 08 03:22:22.486833 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:22.486798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:22.487048 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:22.487024 2576 patch_prober.go:28] interesting pod/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 08 03:22:22.487098 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:22.487080 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 08 03:22:23.486648 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:23.486619 2576 patch_prober.go:28] interesting pod/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 08 03:22:23.487090 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:23.487068 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 08 03:22:24.487133 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:24.487094 2576 patch_prober.go:28] interesting pod/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 08 03:22:24.487605 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:24.487152 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 08 03:22:24.712049 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:24.712009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:25.054572 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:25.054543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:25.253066 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:25.253036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:25.354259 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:25.354181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:25.486627 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:25.486599 2576 patch_prober.go:28] interesting pod/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 08 03:22:25.486790 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:25.486663 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 08 03:22:25.564817 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:25.564783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:26.490185 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:26.490156 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:26.490482 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:26.490448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:26.491224 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:26.491206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:27.230474 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:27.230444 2576 patch_prober.go:28] interesting pod/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 03:22:27.230845 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:27.230493 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 03:22:27.594961 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:27.594884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:27.696211 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:27.696185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:29.289762 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:29.289730 2576 generic.go:356] "Generic (PLEG): container finished" podID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerID="8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96" exitCode=0 Jul 08 03:22:29.290209 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:29.289805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" event={"ID":"bba0f02c-bdf2-4e64-b2a9-471f81510401","Type":"ContainerDied","Data":"8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96"} Jul 08 03:22:29.290209 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:29.290090 2576 scope.go:117] "RemoveContainer" containerID="8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96" Jul 08 03:22:29.291017 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:29.290993 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:29.291071 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:29.291058 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:29.291110 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:29.291089 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:30.299651 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:30.299608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" event={"ID":"bba0f02c-bdf2-4e64-b2a9-471f81510401","Type":"ContainerStarted","Data":"5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8"} Jul 08 03:22:30.635118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:30.635086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:30.635298 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:30.635280 2576 patch_prober.go:28] interesting pod/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 03:22:30.635344 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:30.635322 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 03:22:30.911423 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:30.911319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:31.012976 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.012880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:31.160803 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.160772 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr"] Jul 08 03:22:31.161009 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.160963 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" containerID="cri-o://ed9bac5e9dced14814d4e0066ccd36c9513db40f458fa55db991309d1cf660c0" gracePeriod=30 Jul 08 03:22:31.181044 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.180967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:31.606984 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.606884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:31.635446 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.635415 2576 patch_prober.go:28] interesting pod/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 03:22:31.635569 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.635469 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 03:22:31.951937 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:31.951904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:32.055920 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:32.055874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:32.155083 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:32.155036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:32.328288 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:32.328186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:32.634895 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:32.634867 2576 patch_prober.go:28] interesting pod/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 03:22:32.635246 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:32.634926 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 03:22:33.635007 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:33.634974 2576 patch_prober.go:28] interesting pod/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 03:22:33.635405 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:33.635035 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 03:22:34.638109 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:34.638083 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:34.638483 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:34.638464 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:34.639216 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:34.639193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:36.339121 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.339092 2576 generic.go:356] "Generic (PLEG): container finished" podID="dfe66094-f26f-48f8-acc6-378766fa7854" containerID="ed9bac5e9dced14814d4e0066ccd36c9513db40f458fa55db991309d1cf660c0" exitCode=0 Jul 08 03:22:36.339517 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.339167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" event={"ID":"dfe66094-f26f-48f8-acc6-378766fa7854","Type":"ContainerDied","Data":"ed9bac5e9dced14814d4e0066ccd36c9513db40f458fa55db991309d1cf660c0"} Jul 08 03:22:36.339517 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.339255 2576 scope.go:117] "RemoveContainer" containerID="2f8cd28d66d4d58a76eda0e38565b598179b93fe69fd2742ef081d0720eed9f9" Jul 08 03:22:36.408196 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.408169 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:36.461242 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461186 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe66094-f26f-48f8-acc6-378766fa7854-istio-podinfo\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461242 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-credential-socket\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461407 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461294 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe66094-f26f-48f8-acc6-378766fa7854-istiod-ca-cert\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461407 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-data\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461407 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461367 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jmx7\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-kube-api-access-9jmx7\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461407 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-istio-token\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461572 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461422 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-certs\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461572 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461439 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-socket\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461572 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461480 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-envoy\") pod \"dfe66094-f26f-48f8-acc6-378766fa7854\" (UID: \"dfe66094-f26f-48f8-acc6-378766fa7854\") " Jul 08 03:22:36.461724 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461592 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:36.461797 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461769 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe66094-f26f-48f8-acc6-378766fa7854-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:22:36.461797 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461775 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:36.461898 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.461822 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.462387 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.462126 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:36.462509 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.462458 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:36.463515 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.463490 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe66094-f26f-48f8-acc6-378766fa7854-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:22:36.463613 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.463590 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:36.463727 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.463708 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:22:36.463786 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.463771 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-kube-api-access-9jmx7" (OuterVolumeSpecName: "kube-api-access-9jmx7") pod "dfe66094-f26f-48f8-acc6-378766fa7854" (UID: "dfe66094-f26f-48f8-acc6-378766fa7854"). InnerVolumeSpecName "kube-api-access-9jmx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:22:36.562721 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562698 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe66094-f26f-48f8-acc6-378766fa7854-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562721 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562720 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562729 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jmx7\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-kube-api-access-9jmx7\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562738 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe66094-f26f-48f8-acc6-378766fa7854-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562748 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562755 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562762 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe66094-f26f-48f8-acc6-378766fa7854-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:36.562857 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:36.562770 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe66094-f26f-48f8-acc6-378766fa7854-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:37.344707 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.344680 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" Jul 08 03:22:37.345112 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.344729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr" event={"ID":"dfe66094-f26f-48f8-acc6-378766fa7854","Type":"ContainerDied","Data":"e998759ddf9c582bfeefe7bd413e75cbbb7d473f939c0b452d1d7928cd9262d5"} Jul 08 03:22:37.345112 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.344788 2576 scope.go:117] "RemoveContainer" containerID="ed9bac5e9dced14814d4e0066ccd36c9513db40f458fa55db991309d1cf660c0" Jul 08 03:22:37.367145 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.367113 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr"] Jul 08 03:22:37.368970 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.368948 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ed1785-openshift-default-764b647c5-2vnkr"] Jul 08 03:22:37.622094 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.622031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:37.723581 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.723550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:37.935970 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.935943 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr"] Jul 08 03:22:37.936123 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.936102 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" containerID="cri-o://5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8" gracePeriod=30 Jul 08 03:22:37.940228 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:37.940173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:38.277955 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.277884 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" path="/var/lib/kubelet/pods/dfe66094-f26f-48f8-acc6-378766fa7854/volumes" Jul 08 03:22:38.441569 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.441539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:38.752707 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.752673 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx"] Jul 08 03:22:38.753152 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753137 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" Jul 08 03:22:38.753203 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753155 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" Jul 08 03:22:38.753203 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753175 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" Jul 08 03:22:38.753203 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753183 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" Jul 08 03:22:38.753305 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753281 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" Jul 08 03:22:38.753305 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753291 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe66094-f26f-48f8-acc6-378766fa7854" containerName="istio-proxy" Jul 08 03:22:38.753305 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.753303 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec7fe3d-6ca8-4448-9767-38c4027343d4" containerName="istio-proxy" Jul 08 03:22:38.756647 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.756623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.759218 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.759197 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-34dd30-openshift-default-dockercfg-9mm7f\"" Jul 08 03:22:38.767809 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.767787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx"] Jul 08 03:22:38.780312 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-socket\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780428 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvw7\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-kube-api-access-tpvw7\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780477 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-envoy\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780477 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-data\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780554 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-credential-socket\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780554 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-podinfo\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780634 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istiod-ca-cert\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780634 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-token\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.780634 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.780619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-certs\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.790174 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.790154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:38.881173 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvw7\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-kube-api-access-tpvw7\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-envoy\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-data\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-credential-socket\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881386 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-podinfo\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881606 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istiod-ca-cert\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881606 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-token\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881606 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-certs\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881606 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-socket\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.881813 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-data\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.882073 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.882009 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-credential-socket\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.882073 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.881993 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-socket\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.882217 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.882107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istiod-ca-cert\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.882217 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.882127 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-certs\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.883740 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.883723 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-envoy\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.883825 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.883784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-podinfo\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.890655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.890630 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-token\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.890927 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.890906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvw7\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-kube-api-access-tpvw7\") pod \"e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:38.909932 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:38.909894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:39.068088 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.068026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:39.210636 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.210612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx"] Jul 08 03:22:39.213078 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:22:39.213046 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16b0359_a5fa_47db_b0cd_3605cc9b7c51.slice/crio-74a69bd5b756fcdf579e3c46e5ada2ac684d6fa6e77f8d5ecd1363752c1dc93a WatchSource:0}: Error finding container 74a69bd5b756fcdf579e3c46e5ada2ac684d6fa6e77f8d5ecd1363752c1dc93a: Status 404 returned error can't find the container with id 74a69bd5b756fcdf579e3c46e5ada2ac684d6fa6e77f8d5ecd1363752c1dc93a Jul 08 03:22:39.215321 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.215258 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:39.215443 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.215349 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:39.215443 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.215400 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:39.357047 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.356961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" event={"ID":"b16b0359-a5fa-47db-b0cd-3605cc9b7c51","Type":"ContainerStarted","Data":"a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8"} Jul 08 03:22:39.357047 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.357014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" event={"ID":"b16b0359-a5fa-47db-b0cd-3605cc9b7c51","Type":"ContainerStarted","Data":"74a69bd5b756fcdf579e3c46e5ada2ac684d6fa6e77f8d5ecd1363752c1dc93a"} Jul 08 03:22:39.377506 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:39.377442 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podStartSLOduration=1.37742307 podStartE2EDuration="1.37742307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:22:39.376888486 +0000 UTC m=+2549.655265689" watchObservedRunningTime="2026-07-08 03:22:39.37742307 +0000 UTC m=+2549.655800272" Jul 08 03:22:40.068255 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:40.068205 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:40.072730 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:40.072705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:40.363990 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:40.363912 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:40.364716 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:40.364697 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:41.276315 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:41.276285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:41.375686 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:41.375657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:41.495121 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:41.495066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:41.742108 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:41.742076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:41.928621 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:41.928593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:42.029904 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:42.029834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:43.181336 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.181302 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:43.219337 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219280 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-envoy\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219337 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219307 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-data\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219538 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219393 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-certs\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219538 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-socket\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219538 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219434 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-token\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219538 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219488 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-credential-socket\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219538 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219537 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-podinfo\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwdmr\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-kube-api-access-pwdmr\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219595 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba0f02c-bdf2-4e64-b2a9-471f81510401-istiod-ca-cert\") pod \"bba0f02c-bdf2-4e64-b2a9-471f81510401\" (UID: \"bba0f02c-bdf2-4e64-b2a9-471f81510401\") " Jul 08 03:22:43.219793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219684 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:43.219793 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.219778 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:43.220049 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.220014 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.220049 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.220040 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.220450 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.220412 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-data" (OuterVolumeSpecName: "istio-data") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:43.220583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.220560 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba0f02c-bdf2-4e64-b2a9-471f81510401-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:22:43.220762 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.220737 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:43.222273 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.222246 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-token" (OuterVolumeSpecName: "istio-token") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:22:43.222404 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.222280 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-kube-api-access-pwdmr" (OuterVolumeSpecName: "kube-api-access-pwdmr") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "kube-api-access-pwdmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:22:43.222473 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.222457 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:22:43.222547 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.222471 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bba0f02c-bdf2-4e64-b2a9-471f81510401" (UID: "bba0f02c-bdf2-4e64-b2a9-471f81510401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:22:43.321311 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321286 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.321311 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321310 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.321473 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321320 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.321473 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321329 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdmr\" (UniqueName: \"kubernetes.io/projected/bba0f02c-bdf2-4e64-b2a9-471f81510401-kube-api-access-pwdmr\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.321473 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321337 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba0f02c-bdf2-4e64-b2a9-471f81510401-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.321473 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321346 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.321473 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.321376 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba0f02c-bdf2-4e64-b2a9-471f81510401-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:22:43.379575 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.379552 2576 generic.go:356] "Generic (PLEG): container finished" podID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerID="5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8" exitCode=0 Jul 08 03:22:43.379678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.379599 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" Jul 08 03:22:43.379678 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.379645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" event={"ID":"bba0f02c-bdf2-4e64-b2a9-471f81510401","Type":"ContainerDied","Data":"5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8"} Jul 08 03:22:43.379792 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.379703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr" event={"ID":"bba0f02c-bdf2-4e64-b2a9-471f81510401","Type":"ContainerDied","Data":"aae511b07e7cfa9575b875a1b451fa6519e8120819eeb1ce7d0a1afbad7443ae"} Jul 08 03:22:43.379792 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.379736 2576 scope.go:117] "RemoveContainer" containerID="5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8" Jul 08 03:22:43.388477 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.388460 2576 scope.go:117] "RemoveContainer" containerID="8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96" Jul 08 03:22:43.396760 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.396744 2576 scope.go:117] "RemoveContainer" containerID="5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8" Jul 08 03:22:43.396997 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:22:43.396978 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8\": container with ID starting with 5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8 not found: ID does not exist" containerID="5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8" Jul 08 03:22:43.397069 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.397008 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8"} err="failed to get container status \"5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8\": rpc error: code = NotFound desc = could not find container \"5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8\": container with ID starting with 5c159c2ce5f904f8436c6550007b2cc0381914eb87491cac181be029de87c2a8 not found: ID does not exist" Jul 08 03:22:43.397069 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.397031 2576 scope.go:117] "RemoveContainer" containerID="8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96" Jul 08 03:22:43.397267 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:22:43.397248 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96\": container with ID starting with 8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96 not found: ID does not exist" containerID="8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96" Jul 08 03:22:43.397306 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.397276 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96"} err="failed to get container status \"8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96\": rpc error: code = NotFound desc = could not find container \"8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96\": container with ID starting with 8d581a37d1e1d40545211180fe551682c9574fd4190ea1178f1bdf0d2d34df96 not found: ID does not exist" Jul 08 03:22:43.400326 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.400306 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr"] Jul 08 03:22:43.404657 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:43.404634 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a4e50e-openshift-default-5bbdc5d9c7-7mqvr"] Jul 08 03:22:44.277904 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:44.277872 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" path="/var/lib/kubelet/pods/bba0f02c-bdf2-4e64-b2a9-471f81510401/volumes" Jul 08 03:22:45.740346 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:45.740307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:45.793424 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:45.793395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:46.096588 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.096518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:46.394580 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.394549 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j"] Jul 08 03:22:46.395109 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.395084 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" Jul 08 03:22:46.395109 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.395103 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" Jul 08 03:22:46.395289 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.395228 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" Jul 08 03:22:46.395289 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.395240 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" Jul 08 03:22:46.397496 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.397479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.400297 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.400278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6493fa-openshift-default-dockercfg-xssgk\"" Jul 08 03:22:46.407323 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.407303 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j"] Jul 08 03:22:46.425640 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.425604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:46.443759 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-envoy\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.443851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-token\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.443851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istiod-ca-cert\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.443851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-data\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.443952 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-podinfo\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.443987 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-socket\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.444021 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.443983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkrv\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-kube-api-access-rjkrv\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.444064 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.444050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-certs\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.444118 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.444105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-credential-socket\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.544714 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.544686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-credential-socket\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.544910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.544729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-envoy\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.544910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.544747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-token\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.544910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.544770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istiod-ca-cert\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.544910 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.544797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-data\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.544939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-podinfo\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-socket\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545140 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkrv\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-kube-api-access-rjkrv\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-credential-socket\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-certs\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545332 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-data\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545558 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-socket\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545558 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545432 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-certs\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.545808 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.545791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istiod-ca-cert\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.547880 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.547847 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-envoy\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.547991 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.547948 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-podinfo\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.552904 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.552881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-token\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.553192 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.553171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkrv\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-kube-api-access-rjkrv\") pod \"e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.625544 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.625505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:46.710208 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.710135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:46.845097 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.845067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j"] Jul 08 03:22:46.846474 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:22:46.846449 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0ee627_a96d_48a8_9d58_44e95ecb4079.slice/crio-a1f383e969ae1939f23e935971061ca94568c3ba751590800a7b682380571069 WatchSource:0}: Error finding container a1f383e969ae1939f23e935971061ca94568c3ba751590800a7b682380571069: Status 404 returned error can't find the container with id a1f383e969ae1939f23e935971061ca94568c3ba751590800a7b682380571069 Jul 08 03:22:46.848566 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.848531 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:46.848655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.848600 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:46.848655 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:46.848627 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:47.401672 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:47.401643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" event={"ID":"fb0ee627-a96d-48a8-9d58-44e95ecb4079","Type":"ContainerStarted","Data":"0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7"} Jul 08 03:22:47.401836 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:47.401684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" event={"ID":"fb0ee627-a96d-48a8-9d58-44e95ecb4079","Type":"ContainerStarted","Data":"a1f383e969ae1939f23e935971061ca94568c3ba751590800a7b682380571069"} Jul 08 03:22:47.421880 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:47.421833 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podStartSLOduration=1.421814079 podStartE2EDuration="1.421814079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:22:47.419948838 +0000 UTC m=+2557.698326053" watchObservedRunningTime="2026-07-08 03:22:47.421814079 +0000 UTC m=+2557.700191279" Jul 08 03:22:47.710849 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:47.710769 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:47.715218 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:47.715198 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:48.405650 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:48.405616 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:48.406509 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:48.406492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:22:49.294273 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:49.294206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:49.386832 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:49.386782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:49.462529 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:49.462463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:49.734219 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:49.734159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:49.856260 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:49.856207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:49.918035 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:49.918007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:53.783546 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:53.783509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:54.188565 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:54.188532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:56.405648 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:56.405611 2576 patch_prober.go:28] interesting pod/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 03:22:56.406234 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:56.406199 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 03:22:57.454083 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:57.454048 2576 generic.go:356] "Generic (PLEG): container finished" podID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerID="a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8" exitCode=0 Jul 08 03:22:57.454636 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:57.454124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" event={"ID":"b16b0359-a5fa-47db-b0cd-3605cc9b7c51","Type":"ContainerDied","Data":"a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8"} Jul 08 03:22:57.454720 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:57.454641 2576 scope.go:117] "RemoveContainer" containerID="a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8" Jul 08 03:22:57.455708 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:57.455676 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:57.455781 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:57.455765 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:57.455829 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:57.455813 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:22:58.460418 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:58.460386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" event={"ID":"b16b0359-a5fa-47db-b0cd-3605cc9b7c51","Type":"ContainerStarted","Data":"8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39"} Jul 08 03:22:59.068791 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:59.068757 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:22:59.068979 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:59.068856 2576 patch_prober.go:28] interesting pod/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 03:22:59.068979 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:22:59.068902 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 03:23:00.068392 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:00.068343 2576 patch_prober.go:28] interesting pod/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 03:23:00.068831 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:00.068411 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 03:23:01.069167 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:01.069134 2576 patch_prober.go:28] interesting pod/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 03:23:01.069545 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:01.069186 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 03:23:02.068873 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:02.068829 2576 patch_prober.go:28] interesting pod/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 03:23:02.069077 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:02.068901 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 03:23:03.072674 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:03.072637 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:23:03.073126 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:03.072924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:23:03.073695 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:03.073679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:23:03.492509 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:03.492483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:23:03.592238 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:03.592179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:23:04.281735 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.281706 2576 patch_prober.go:28] interesting pod/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:38778->10.133.0.48:15021: read: connection reset by peer" start-of-body= Jul 08 03:23:04.282132 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.281769 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:38778->10.133.0.48:15021: read: connection reset by peer" Jul 08 03:23:04.492347 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.492294 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerID="0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7" exitCode=0 Jul 08 03:23:04.492476 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.492390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" event={"ID":"fb0ee627-a96d-48a8-9d58-44e95ecb4079","Type":"ContainerDied","Data":"0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7"} Jul 08 03:23:04.492929 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.492911 2576 scope.go:117] "RemoveContainer" containerID="0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7" Jul 08 03:23:04.493883 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.493854 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:23:04.493979 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.493944 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:23:04.493979 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:04.493976 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 08 03:23:05.502288 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:05.502255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" event={"ID":"fb0ee627-a96d-48a8-9d58-44e95ecb4079","Type":"ContainerStarted","Data":"caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28"} Jul 08 03:23:05.711107 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:05.711074 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:23:05.711505 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:05.711418 2576 patch_prober.go:28] interesting pod/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 03:23:05.711693 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:05.711672 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 03:23:06.710694 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:06.710660 2576 patch_prober.go:28] interesting pod/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 03:23:06.711257 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:06.711223 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 03:23:07.711084 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:07.711043 2576 patch_prober.go:28] interesting pod/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 03:23:07.711588 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:07.711118 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 03:23:08.711374 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:08.711319 2576 patch_prober.go:28] interesting pod/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 03:23:08.711844 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:08.711418 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 03:23:09.714090 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:09.714063 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:23:09.714585 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:09.714452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:23:09.715061 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:09.715042 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:23:11.531560 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:11.531520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:23:11.631851 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:23:11.631826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:10.357200 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:25:10.357126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:25:10.366992 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:25:10.366975 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:25:58.767017 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:25:58.766988 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:00.193290 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:00.193244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:04.826525 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:04.826489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:04.890458 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:04.890431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:05.089651 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:05.089569 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx"] Jul 08 03:26:05.089819 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:05.089796 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" containerID="cri-o://8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39" gracePeriod=30 Jul 08 03:26:05.114246 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:05.114216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:05.519467 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:05.519421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:05.908181 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:05.908136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:06.008130 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:06.008098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:06.109136 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:06.109097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:06.211796 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:06.211709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:10.333081 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.333059 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:26:10.355090 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.355031 2576 generic.go:356] "Generic (PLEG): container finished" podID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerID="8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39" exitCode=0 Jul 08 03:26:10.355090 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.355089 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" Jul 08 03:26:10.355267 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.355119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" event={"ID":"b16b0359-a5fa-47db-b0cd-3605cc9b7c51","Type":"ContainerDied","Data":"8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39"} Jul 08 03:26:10.355267 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.355177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx" event={"ID":"b16b0359-a5fa-47db-b0cd-3605cc9b7c51","Type":"ContainerDied","Data":"74a69bd5b756fcdf579e3c46e5ada2ac684d6fa6e77f8d5ecd1363752c1dc93a"} Jul 08 03:26:10.355267 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.355202 2576 scope.go:117] "RemoveContainer" containerID="8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39" Jul 08 03:26:10.364784 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.364767 2576 scope.go:117] "RemoveContainer" containerID="a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8" Jul 08 03:26:10.372249 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.372228 2576 scope.go:117] "RemoveContainer" containerID="8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39" Jul 08 03:26:10.372528 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:26:10.372509 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39\": container with ID starting with 8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39 not found: ID does not exist" containerID="8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39" Jul 08 03:26:10.372597 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.372537 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39"} err="failed to get container status \"8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39\": rpc error: code = NotFound desc = could not find container \"8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39\": container with ID starting with 8dc69264c211ebcc0e5ea4462e33cccb5ae474da3408739f0fcbac2ca44c5e39 not found: ID does not exist" Jul 08 03:26:10.372597 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.372554 2576 scope.go:117] "RemoveContainer" containerID="a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8" Jul 08 03:26:10.372802 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:26:10.372775 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8\": container with ID starting with a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8 not found: ID does not exist" containerID="a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8" Jul 08 03:26:10.372845 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.372800 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8"} err="failed to get container status \"a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8\": rpc error: code = NotFound desc = could not find container \"a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8\": container with ID starting with a5836955013488491092c72ec919f8ad7bd34b629efa15905fd6fe60aea19fd8 not found: ID does not exist" Jul 08 03:26:10.524482 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524460 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istiod-ca-cert\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-envoy\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524513 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpvw7\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-kube-api-access-tpvw7\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524583 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-data\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524744 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524590 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-podinfo\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524744 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524618 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-token\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524744 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524688 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-socket\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524911 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524751 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-certs\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524911 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524799 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-credential-socket\") pod \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\" (UID: \"b16b0359-a5fa-47db-b0cd-3605cc9b7c51\") " Jul 08 03:26:10.524911 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.524808 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:26:10.525077 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.525063 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:10.525135 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.525073 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:10.525135 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.525102 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.525490 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.525466 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:10.525649 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.525631 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-data" (OuterVolumeSpecName: "istio-data") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:10.526651 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.526624 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:10.527031 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.527014 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-token" (OuterVolumeSpecName: "istio-token") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:26:10.527227 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.527199 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:26:10.527306 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.527202 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-kube-api-access-tpvw7" (OuterVolumeSpecName: "kube-api-access-tpvw7") pod "b16b0359-a5fa-47db-b0cd-3605cc9b7c51" (UID: "b16b0359-a5fa-47db-b0cd-3605cc9b7c51"). InnerVolumeSpecName "kube-api-access-tpvw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:26:10.625838 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625817 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625838 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625837 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625845 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625853 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625862 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625870 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvw7\" (UniqueName: \"kubernetes.io/projected/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-kube-api-access-tpvw7\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625878 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.625969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.625886 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16b0359-a5fa-47db-b0cd-3605cc9b7c51-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:10.676306 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.676281 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx"] Jul 08 03:26:10.688557 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:10.688533 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-34dd30-openshift-default-6cd784f7cf-fnprx"] Jul 08 03:26:11.771153 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:11.771125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:11.871046 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:11.870992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:12.038169 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:12.038097 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j"] Jul 08 03:26:12.038433 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:12.038401 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" containerID="cri-o://caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28" gracePeriod=30 Jul 08 03:26:12.057242 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:12.057217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:12.278461 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:12.278431 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" path="/var/lib/kubelet/pods/b16b0359-a5fa-47db-b0cd-3605cc9b7c51/volumes" Jul 08 03:26:17.288321 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.288298 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:26:17.372115 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372056 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-socket\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372115 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372104 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-certs\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372295 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372144 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-envoy\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372295 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-podinfo\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372295 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372185 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istiod-ca-cert\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372295 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-credential-socket\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372295 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjkrv\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-kube-api-access-rjkrv\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372295 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372293 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-data\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372323 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-token\") pod \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\" (UID: \"fb0ee627-a96d-48a8-9d58-44e95ecb4079\") " Jul 08 03:26:17.372612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372338 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:17.372612 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372574 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:17.372782 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372723 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.372782 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372743 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-workload-certs\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.372782 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372745 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:26:17.373036 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.372999 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:17.373271 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.373252 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:17.374538 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.374517 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-kube-api-access-rjkrv" (OuterVolumeSpecName: "kube-api-access-rjkrv") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "kube-api-access-rjkrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:26:17.374637 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.374546 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:26:17.374891 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.374869 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:17.374988 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.374934 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb0ee627-a96d-48a8-9d58-44e95ecb4079" (UID: "fb0ee627-a96d-48a8-9d58-44e95ecb4079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:26:17.388536 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.388509 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerID="caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28" exitCode=0 Jul 08 03:26:17.388635 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.388553 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" Jul 08 03:26:17.388635 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.388583 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" event={"ID":"fb0ee627-a96d-48a8-9d58-44e95ecb4079","Type":"ContainerDied","Data":"caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28"} Jul 08 03:26:17.388717 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.388654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j" event={"ID":"fb0ee627-a96d-48a8-9d58-44e95ecb4079","Type":"ContainerDied","Data":"a1f383e969ae1939f23e935971061ca94568c3ba751590800a7b682380571069"} Jul 08 03:26:17.388717 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.388676 2576 scope.go:117] "RemoveContainer" containerID="caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28" Jul 08 03:26:17.397629 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.397613 2576 scope.go:117] "RemoveContainer" containerID="0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7" Jul 08 03:26:17.405614 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.405599 2576 scope.go:117] "RemoveContainer" containerID="caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28" Jul 08 03:26:17.405843 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:26:17.405826 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28\": container with ID starting with caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28 not found: ID does not exist" containerID="caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28" Jul 08 03:26:17.405884 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.405849 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28"} err="failed to get container status \"caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28\": rpc error: code = NotFound desc = could not find container \"caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28\": container with ID starting with caa359072f5ef1077667fa198d59242f662ef150fa42994c7ecab7ec1ebcda28 not found: ID does not exist" Jul 08 03:26:17.405884 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.405865 2576 scope.go:117] "RemoveContainer" containerID="0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7" Jul 08 03:26:17.406071 ip-10-0-128-107 kubenswrapper[2576]: E0708 03:26:17.406055 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7\": container with ID starting with 0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7 not found: ID does not exist" containerID="0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7" Jul 08 03:26:17.406114 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.406074 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7"} err="failed to get container status \"0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7\": rpc error: code = NotFound desc = could not find container \"0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7\": container with ID starting with 0f4023284356b8acdea072ba6d36d65d7d2e67c55b5e09c530033278945abef7 not found: ID does not exist" Jul 08 03:26:17.409164 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.409143 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j"] Jul 08 03:26:17.413471 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.413454 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6493fa-openshift-default-769db49659-wcm8j"] Jul 08 03:26:17.473759 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473736 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-envoy\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.473759 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473756 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-podinfo\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.473871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473765 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istiod-ca-cert\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.473871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473774 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-credential-socket\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.473871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473783 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkrv\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-kube-api-access-rjkrv\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.473871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473791 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-data\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.473871 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.473799 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0ee627-a96d-48a8-9d58-44e95ecb4079-istio-token\") on node \"ip-10-0-128-107.ec2.internal\" DevicePath \"\"" Jul 08 03:26:17.658153 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:17.658122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-664c944698-jhg88_f7187e5d-e3df-4c6c-8dfe-60fbab125d9b/manager/0.log" Jul 08 03:26:18.002300 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:18.002240 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fbb74d9dd-8ncml_50770948-121c-40b6-a313-7d461aebc60b/copy-manifests/0.log" Jul 08 03:26:18.020469 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:18.020449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fbb74d9dd-8ncml_50770948-121c-40b6-a313-7d461aebc60b/manager/0.log" Jul 08 03:26:18.279015 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:18.278952 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" path="/var/lib/kubelet/pods/fb0ee627-a96d-48a8-9d58-44e95ecb4079/volumes" Jul 08 03:26:20.736003 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:20.735969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jtvbk_90bd1918-d796-4039-a592-ccefd26d35c7/istio-proxy/0.log" Jul 08 03:26:20.860733 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:20.860704 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jktwk_c9c45844-eeeb-4fe4-9454-27cfce5ddfee/discovery/0.log" Jul 08 03:26:21.555165 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:21.555141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f8b685459-ftl64_ccd97aff-9a22-4b34-84c2-3eb2ea9596a4/router/0.log" Jul 08 03:26:22.248663 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:22.248636 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwxz2_89c16bbc-ca04-4941-bc61-b672e51b02dc/main/0.log" Jul 08 03:26:26.694947 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:26.694917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:26.791929 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:26.791902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:30.075838 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:30.075810 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w26sd_e920314d-549a-4cd1-8160-4decdd56ad45/global-pull-secret-syncer/0.log" Jul 08 03:26:30.151721 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:30.151692 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" Jul 08 03:26:30.271097 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:30.271075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-107.ec2.internal_f42daac05cfc00507a8ce4234e1a889f/haproxy/0.log" Jul 08 03:26:37.315578 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.315549 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/alertmanager/0.log" Jul 08 03:26:37.354914 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.354892 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/config-reloader/0.log" Jul 08 03:26:37.391893 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.391872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/kube-rbac-proxy-web/0.log" Jul 08 03:26:37.417566 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.417497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/kube-rbac-proxy/0.log" Jul 08 03:26:37.446987 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.446965 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/kube-rbac-proxy-metric/0.log" Jul 08 03:26:37.481669 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.481648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/prom-label-proxy/0.log" Jul 08 03:26:37.506832 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.506816 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96c0734c-c1cb-46cc-945d-3f7352a37a33/init-config-reloader/0.log" Jul 08 03:26:37.769278 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.769214 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4dxkb_8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0/node-exporter/0.log" Jul 08 03:26:37.793461 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.793440 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4dxkb_8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0/kube-rbac-proxy/0.log" Jul 08 03:26:37.823915 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:37.823898 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4dxkb_8ea83bf1-92b3-4f9a-836b-d3c07dc09cd0/init-textfile/0.log" Jul 08 03:26:38.060435 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.060379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-l5xdd_3ac39c34-7009-471e-832e-5c944fe3a1d0/kube-rbac-proxy-main/0.log" Jul 08 03:26:38.090807 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.090792 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-l5xdd_3ac39c34-7009-471e-832e-5c944fe3a1d0/kube-rbac-proxy-self/0.log" Jul 08 03:26:38.114876 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.114853 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-l5xdd_3ac39c34-7009-471e-832e-5c944fe3a1d0/openshift-state-metrics/0.log" Jul 08 03:26:38.431328 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.431297 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-p7nrx_27ccdff5-d3ff-49e0-9643-b01f43e0946f/prometheus-operator-admission-webhook/0.log" Jul 08 03:26:38.460989 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.460969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9bc669fb7-grtwd_f4ec95c0-1f4f-4666-935b-e3104fa2bafe/telemeter-client/0.log" Jul 08 03:26:38.483037 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.483000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9bc669fb7-grtwd_f4ec95c0-1f4f-4666-935b-e3104fa2bafe/reload/0.log" Jul 08 03:26:38.505602 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:38.505584 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9bc669fb7-grtwd_f4ec95c0-1f4f-4666-935b-e3104fa2bafe/kube-rbac-proxy/0.log" Jul 08 03:26:39.949601 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949570 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9"] Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949881 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949892 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949904 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949909 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba0f02c-bdf2-4e64-b2a9-471f81510401" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949916 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949922 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949932 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" Jul 08 03:26:39.949950 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949936 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" Jul 08 03:26:39.950187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.949994 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" Jul 08 03:26:39.950187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.950003 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0ee627-a96d-48a8-9d58-44e95ecb4079" containerName="istio-proxy" Jul 08 03:26:39.950187 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.950009 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b16b0359-a5fa-47db-b0cd-3605cc9b7c51" containerName="istio-proxy" Jul 08 03:26:39.954003 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.953984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:39.956450 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.956435 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wbm7l\"/\"kube-root-ca.crt\"" Jul 08 03:26:39.957245 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.957220 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wbm7l\"/\"default-dockercfg-q7m95\"" Jul 08 03:26:39.957348 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.957224 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wbm7l\"/\"openshift-service-ca.crt\"" Jul 08 03:26:39.962969 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:39.962941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9"] Jul 08 03:26:40.043989 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.043962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-sys\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.044092 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.043993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-podres\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.044092 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.044018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q44c\" (UniqueName: \"kubernetes.io/projected/460ef799-6b6a-44a7-9141-d4c2eb8153f3-kube-api-access-2q44c\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.044181 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.044089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-lib-modules\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.044181 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.044117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-proc\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.144974 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.144950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-sys\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145074 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.144976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-podres\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145074 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.144993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q44c\" (UniqueName: \"kubernetes.io/projected/460ef799-6b6a-44a7-9141-d4c2eb8153f3-kube-api-access-2q44c\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145074 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.145009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-lib-modules\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145074 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.145026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-proc\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145251 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.145125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-proc\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145251 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.145129 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-lib-modules\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145251 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.145135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-podres\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.145251 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.145125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/460ef799-6b6a-44a7-9141-d4c2eb8153f3-sys\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.153785 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.153766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q44c\" (UniqueName: \"kubernetes.io/projected/460ef799-6b6a-44a7-9141-d4c2eb8153f3-kube-api-access-2q44c\") pod \"perf-node-gather-daemonset-v6dg9\" (UID: \"460ef799-6b6a-44a7-9141-d4c2eb8153f3\") " pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.264672 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.264618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:40.396887 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.396855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9"] Jul 08 03:26:40.399867 ip-10-0-128-107 kubenswrapper[2576]: W0708 03:26:40.399833 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod460ef799_6b6a_44a7_9141_d4c2eb8153f3.slice/crio-b29f419f95725be55e09c0465bab9be23082f4fe064a421d4e7f537d3c02de42 WatchSource:0}: Error finding container b29f419f95725be55e09c0465bab9be23082f4fe064a421d4e7f537d3c02de42: Status 404 returned error can't find the container with id b29f419f95725be55e09c0465bab9be23082f4fe064a421d4e7f537d3c02de42 Jul 08 03:26:40.490574 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.490541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" event={"ID":"460ef799-6b6a-44a7-9141-d4c2eb8153f3","Type":"ContainerStarted","Data":"b29f419f95725be55e09c0465bab9be23082f4fe064a421d4e7f537d3c02de42"} Jul 08 03:26:40.579363 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.579337 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/1.log" Jul 08 03:26:40.588014 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:40.587994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-786pp_959b648d-524d-4fb4-961b-b32f222717ee/console-operator/2.log" Jul 08 03:26:41.496387 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:41.496341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" event={"ID":"460ef799-6b6a-44a7-9141-d4c2eb8153f3","Type":"ContainerStarted","Data":"fb004b9376206ea685e46f9e9adbb28c2897c56122e0e82d4b750822caa1ef2e"} Jul 08 03:26:41.496737 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:41.496437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:41.515294 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:41.515252 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" podStartSLOduration=2.515237575 podStartE2EDuration="2.515237575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 03:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:26:41.512988321 +0000 UTC m=+2791.791365523" watchObservedRunningTime="2026-07-08 03:26:41.515237575 +0000 UTC m=+2791.793614775" Jul 08 03:26:42.586976 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:42.586945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdrk_69de278a-b53e-46db-88ce-bd314cd744f6/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:42.901248 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:42.901222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tdxfs_a6878310-383d-46ab-b566-005f4981c3f7/dns/0.log" Jul 08 03:26:42.923339 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:42.923315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tdxfs_a6878310-383d-46ab-b566-005f4981c3f7/kube-rbac-proxy/0.log" Jul 08 03:26:42.997420 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:42.997395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5s2w_ba220b2f-2b74-4163-a98f-9975b0a172f5/dns-node-resolver/0.log" Jul 08 03:26:43.764859 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:43.764832 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vfjph_010a311a-d396-497b-9742-78f12c75dfd8/node-ca/0.log" Jul 08 03:26:44.814750 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:44.814725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jtvbk_90bd1918-d796-4039-a592-ccefd26d35c7/istio-proxy/0.log" Jul 08 03:26:44.998953 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:44.998927 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jktwk_c9c45844-eeeb-4fe4-9454-27cfce5ddfee/discovery/0.log" Jul 08 03:26:45.245742 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:45.245720 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f8b685459-ftl64_ccd97aff-9a22-4b34-84c2-3eb2ea9596a4/router/0.log" Jul 08 03:26:45.951873 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:45.951851 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8q9px_d8462f8e-cbc6-4d23-8c1c-b99e3813824b/serve-healthcheck-canary/0.log" Jul 08 03:26:46.479869 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:46.479834 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wphwj_7b1e0477-6102-4d78-8329-f3792ea19ef5/insights-operator/0.log" Jul 08 03:26:46.482154 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:46.482133 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wphwj_7b1e0477-6102-4d78-8329-f3792ea19ef5/insights-operator/1.log" Jul 08 03:26:46.630549 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:46.630531 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjnfq_ec513da9-d082-4b0a-a6fb-bb83cbb3ff51/kube-rbac-proxy/0.log" Jul 08 03:26:46.653012 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:46.652989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjnfq_ec513da9-d082-4b0a-a6fb-bb83cbb3ff51/exporter/0.log" Jul 08 03:26:46.676495 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:46.676469 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjnfq_ec513da9-d082-4b0a-a6fb-bb83cbb3ff51/extractor/0.log" Jul 08 03:26:47.510115 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:47.510093 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wbm7l/perf-node-gather-daemonset-v6dg9" Jul 08 03:26:49.487239 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:49.487210 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-664c944698-jhg88_f7187e5d-e3df-4c6c-8dfe-60fbab125d9b/manager/0.log" Jul 08 03:26:49.729714 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:49.729686 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fbb74d9dd-8ncml_50770948-121c-40b6-a313-7d461aebc60b/manager/0.log" Jul 08 03:26:49.747443 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:49.747423 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fbb74d9dd-8ncml_50770948-121c-40b6-a313-7d461aebc60b/copy-manifests/0.log" Jul 08 03:26:51.171167 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:51.171140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56954ff8c6-rz6wh_d7b2442b-b69c-402b-9219-57683cc44445/manager/0.log" Jul 08 03:26:51.203387 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:51.203348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jtvbk_90bd1918-d796-4039-a592-ccefd26d35c7/istio-proxy/0.log" Jul 08 03:26:51.391331 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:51.391309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jktwk_c9c45844-eeeb-4fe4-9454-27cfce5ddfee/discovery/0.log" Jul 08 03:26:51.602821 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:51.602790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f8b685459-ftl64_ccd97aff-9a22-4b34-84c2-3eb2ea9596a4/router/0.log" Jul 08 03:26:52.804923 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:52.804894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jtvbk_90bd1918-d796-4039-a592-ccefd26d35c7/istio-proxy/0.log" Jul 08 03:26:52.991838 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:52.991804 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jktwk_c9c45844-eeeb-4fe4-9454-27cfce5ddfee/discovery/0.log" Jul 08 03:26:53.193026 ip-10-0-128-107 kubenswrapper[2576]: I0708 03:26:53.192993 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f8b685459-ftl64_ccd97aff-9a22-4b34-84c2-3eb2ea9596a4/router/0.log"