Jul 23 00:19:29.108738 ip-10-0-139-142 systemd[1]: Starting Kubernetes Kubelet... Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.526776 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534700 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534711 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534714 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534718 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534721 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534724 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534727 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534730 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:19:29.555831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534733 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534737 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534741 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534744 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534747 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534750 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534753 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534756 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534758 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534761 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534764 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534766 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534769 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534776 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534779 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534782 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534785 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534788 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534790 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:19:29.567927 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534793 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534798 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534803 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534806 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534808 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534811 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534813 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534816 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534819 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534821 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534824 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534826 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534829 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534832 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534836 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534838 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534841 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534844 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534847 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534849 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:19:29.570436 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534852 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534854 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534857 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534859 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534862 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534864 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534866 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534869 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534872 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534875 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534877 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534879 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534882 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534884 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534888 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534891 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534893 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534896 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534898 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534901 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:19:29.572354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534903 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534907 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534909 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534912 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534914 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534917 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534919 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534923 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534926 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534928 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534931 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534933 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534936 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534939 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534941 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534944 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534946 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534949 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534951 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534954 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:19:29.574827 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534966 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534969 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534972 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534974 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534977 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534979 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534984 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534987 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534990 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534993 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534996 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.534999 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535702 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535710 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535713 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535716 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535719 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535722 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535725 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:19:29.577483 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535728 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535731 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535734 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535739 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535742 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535745 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535747 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535750 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535753 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535757 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535759 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535762 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535765 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535768 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535772 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535774 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535777 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535779 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535782 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:19:29.580506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535785 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535787 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535790 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535792 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535795 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535798 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535800 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535805 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535808 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535811 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535814 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535817 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535820 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535822 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535825 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535828 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535830 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535833 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535835 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:19:29.582688 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535838 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535841 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535844 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535846 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535849 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535852 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535855 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535857 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535860 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535864 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535867 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535869 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535873 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535876 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535879 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535882 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535885 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535888 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535891 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535893 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:19:29.585682 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535896 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535900 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535902 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535905 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535907 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535910 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535912 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535915 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535917 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535921 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535925 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535928 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535931 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535933 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535936 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535938 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535941 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535944 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535948 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535951 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:19:29.587889 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535954 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535956 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535959 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535962 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535964 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535967 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535970 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535973 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535975 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535977 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535980 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535983 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535985 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.535988 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536515 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536522 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536526 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536530 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536534 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536538 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:19:29.590488 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536542 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536545 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536549 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536552 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536556 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536559 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536562 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536565 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536569 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536572 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536579 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536582 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536585 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536589 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536593 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536596 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536599 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536603 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536607 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536611 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:19:29.592364 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536615 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536617 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536620 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536624 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536627 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536629 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536632 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536634 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536637 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536639 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536644 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536647 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536650 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536653 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536656 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536659 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536663 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536666 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536669 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:19:29.594701 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536672 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536675 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536678 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536682 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536685 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536687 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536690 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536693 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536697 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536699 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536702 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536704 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536708 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536710 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536713 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536715 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536719 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536722 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536725 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:19:29.596575 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536728 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536731 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536734 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536736 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536739 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536741 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536745 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536747 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536750 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536752 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536755 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536758 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536761 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536763 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536766 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536768 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536772 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536774 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536777 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:19:29.598932 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536780 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536782 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536798 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536801 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536804 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536807 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536809 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536812 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536814 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536817 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536821 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536823 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536826 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536829 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536831 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.536835 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536903 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536914 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536921 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536925 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536929 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536933 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 00:19:29.612091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536937 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 00:19:29.608915 ip-10-0-139-142 systemd[1]: Started Kubernetes Kubelet. Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536940 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536944 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536947 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536951 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536954 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536957 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536962 2585 flags.go:64] FLAG: --cgroup-root="" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536965 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536968 2585 flags.go:64] FLAG: --client-ca-file="" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536971 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536974 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536978 2585 flags.go:64] FLAG: --cluster-domain="" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536981 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536985 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536988 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536991 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536995 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.536998 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537001 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537005 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537008 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537011 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537014 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 00:19:29.613835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537018 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537021 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537025 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537028 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537031 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537034 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537037 2585 flags.go:64] FLAG: --enable-server="true" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537040 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537044 2585 flags.go:64] FLAG: --event-burst="100" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537047 2585 flags.go:64] FLAG: --event-qps="50" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537050 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537053 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537056 2585 flags.go:64] FLAG: --eviction-hard="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537060 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537062 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537067 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537072 2585 flags.go:64] FLAG: --eviction-soft="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537076 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537080 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537082 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537085 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537089 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537091 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537095 2585 flags.go:64] FLAG: --feature-gates="" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537099 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 00:19:29.616403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537102 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537105 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537108 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537112 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537115 2585 flags.go:64] FLAG: --help="false" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537118 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-139-142.ec2.internal" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537121 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537124 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537126 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537130 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537133 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537136 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537139 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537141 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537144 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537147 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537150 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537153 2585 flags.go:64] FLAG: --kube-reserved="" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537156 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537158 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537161 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537164 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537169 2585 flags.go:64] FLAG: --lock-file="" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537172 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 00:19:29.631378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537175 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537178 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537183 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537186 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537189 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537192 2585 flags.go:64] FLAG: --logging-format="text" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537195 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537198 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537201 2585 flags.go:64] FLAG: --manifest-url="" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537204 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537208 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537213 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537216 2585 flags.go:64] FLAG: --max-pods="110" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537219 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537222 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537225 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537228 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537231 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537234 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537237 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537244 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537247 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537249 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 00:19:29.848270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537252 2585 flags.go:64] FLAG: --pod-cidr="" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537255 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537260 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537263 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537266 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537269 2585 flags.go:64] FLAG: --port="10250" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537272 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537275 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dbc5c67bf1f0234f" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537279 2585 flags.go:64] FLAG: --qos-reserved="" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537282 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537285 2585 flags.go:64] FLAG: --register-node="true" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537288 2585 flags.go:64] FLAG: --register-with-taints="" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537292 2585 flags.go:64] FLAG: --registry-burst="10" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537311 2585 flags.go:64] FLAG: --registry-qps="5" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537315 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537319 2585 flags.go:64] FLAG: --reserved-memory="" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537324 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537328 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537332 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537335 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537340 2585 flags.go:64] FLAG: --runonce="false" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537344 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537348 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537352 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537356 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 00:19:29.864807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537359 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537363 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537367 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537371 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537374 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537377 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537381 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537384 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537388 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537392 2585 flags.go:64] FLAG: --system-cgroups="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537395 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537402 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537405 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537409 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537414 2585 flags.go:64] FLAG: --tls-min-version="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537419 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537422 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537426 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537429 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537433 2585 flags.go:64] FLAG: --v="2" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537437 2585 flags.go:64] FLAG: --version="false" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537442 2585 flags.go:64] FLAG: --vmodule="" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537446 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.537451 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537592 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:19:29.880568 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537597 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537600 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537604 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537607 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537610 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537612 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537617 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537620 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537623 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537626 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537629 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537631 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537634 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537636 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537639 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537641 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537644 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537646 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537649 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537651 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:19:29.881867 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537654 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537656 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537660 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537663 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537665 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537668 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537670 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537673 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537675 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537678 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537680 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537683 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537685 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537688 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537692 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537695 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537697 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537700 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537702 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537707 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:19:29.882995 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537710 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537713 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537717 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537720 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537725 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537730 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537734 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537738 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537742 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537746 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537750 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537754 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537757 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537761 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537766 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537770 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537774 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537778 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537782 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:19:29.884093 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537786 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537791 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537795 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537799 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537803 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537808 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537812 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537818 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537822 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537826 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537830 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537834 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537838 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537842 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537847 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537851 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537855 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537858 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537862 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537865 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:19:29.885190 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537869 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537875 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537880 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537884 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537889 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537893 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537898 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537903 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537907 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537911 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537914 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537918 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537923 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537926 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537930 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537934 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537938 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537942 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:19:29.886289 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.537946 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.538720 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.546851 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.546866 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.547666 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.547674 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547734 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547739 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547743 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547746 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547749 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547754 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547758 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547761 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:19:29.887159 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547764 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547767 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547770 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547773 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547776 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547779 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547782 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547785 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547788 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547790 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547793 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547795 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547798 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547800 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547803 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547806 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547808 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547811 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547814 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547816 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:19:29.888029 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547819 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547822 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547825 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547827 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547830 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547833 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547836 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547839 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547841 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547844 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547847 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547849 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547852 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547854 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547857 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547859 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547862 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547864 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547868 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547870 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:19:29.888896 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547873 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547875 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547878 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547880 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547883 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547885 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547889 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547892 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547895 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547898 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547900 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547903 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547906 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547908 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547911 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547914 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547917 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547919 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547922 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547924 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:19:29.889770 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547927 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547930 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547932 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547935 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547937 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547940 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547948 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547953 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547956 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547959 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547961 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547964 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547967 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547969 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547972 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547974 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547977 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547979 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547982 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:19:29.890631 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547984 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547987 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547989 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547992 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547994 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547996 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.547999 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548001 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548004 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548006 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548008 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548011 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.548016 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548131 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548136 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:19:29.891512 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548140 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548142 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548145 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548149 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548151 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548156 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548159 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548162 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548164 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548167 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548170 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548173 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548181 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548184 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548186 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548190 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548193 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548196 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548199 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:19:29.892177 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548203 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548207 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548210 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548214 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548217 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548220 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548223 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548226 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548228 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548231 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548233 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548236 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548238 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548241 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548243 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548246 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548248 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548251 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548256 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548258 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:19:29.893069 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548261 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548264 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548266 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548269 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548272 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548275 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548278 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548280 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548283 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548285 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548288 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548291 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548293 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548306 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548309 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548311 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548314 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548316 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548319 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548321 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:19:29.893942 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548324 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548326 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548329 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548331 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548334 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548336 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548339 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548341 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548344 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548346 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548351 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548354 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548356 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548359 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548362 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548364 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548367 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548370 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548373 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548375 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:19:29.895035 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548378 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548380 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548383 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548385 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548388 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548391 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548393 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548396 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548399 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548401 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548404 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548406 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548409 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548411 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548413 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548416 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548418 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:19:29.895905 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:29.548421 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.548426 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.548533 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.550799 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.552516 2585 server.go:1019] "Starting client certificate rotation" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.552728 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.554074 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.556334 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.557478 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.573463 2585 log.go:25] "Validated CRI v1 runtime API" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.577078 2585 log.go:25] "Validated CRI v1 image API" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.578255 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 00:19:29.897131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.580922 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d93c37dd-8b3e-43d0-a966-e952e2e7139c:/dev/nvme0n1p4 e46c9b5a-2669-4e01-b94b-493c34e965f2:/dev/nvme0n1p3] Jul 23 00:19:29.897789 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.580935 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.589198 2585 manager.go:216] Machine: {Timestamp:2026-07-23 00:19:29.587908365 +0000 UTC m=+0.366769877 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098594 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e46c6c43f139ec12027d6f414c89b SystemUUID:ec2e46c6-c43f-139e-c120-27d6f414c89b BootID:7edc8fab-d55b-4936-809b-dc3946b1a425 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:50:e7:28:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:50:e7:28:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:c8:af:19:85:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.589310 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.589397 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.590101 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.590617 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.590644 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 00:19:29.898007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.591418 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.591431 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.591441 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.592237 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.593945 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597536 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597664 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597698 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597710 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597726 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597737 2585 kubelet.go:398] "Adding apiserver pod source" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.597748 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.598116 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.598134 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.600506 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.602400 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.602427 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603175 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603188 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603194 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603199 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603209 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603217 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603316 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603323 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603329 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603334 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603346 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.603375 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.605044 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.605053 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.608068 2585 server.go:1268] "Started kubelet" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.608151 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.608170 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.608237 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.609646 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.610147 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.610213 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.610270 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 00:19:29.900947 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.610367 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.615980 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.617506 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-142.ec2.internal.18c4c33179b159ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-142.ec2.internal,UID:ip-10-0-139-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-142.ec2.internal,},FirstTimestamp:2026-07-23 00:19:29.608038895 +0000 UTC m=+0.386900406,LastTimestamp:2026-07-23 00:19:29.608038895 +0000 UTC m=+0.386900406,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-142.ec2.internal,}" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.619741 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.619752 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.620287 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620532 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620550 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620567 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620573 2585 factory.go:55] Registering systemd factory Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620612 2585 factory.go:223] Registration of the systemd container factory successfully Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620722 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620732 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620858 2585 factory.go:153] Registering CRI-O factory Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620872 2585 factory.go:223] Registration of the crio container factory successfully Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620928 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620950 2585 factory.go:103] Registering Raw factory Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.620965 2585 manager.go:1196] Started watching for new ooms in manager Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.621424 2585 manager.go:349] Starting recovery of all containers Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.623584 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.623670 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.632454 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lpgpx" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.634327 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.636115 2585 manager.go:354] Recovery completed Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.637451 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lpgpx" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.640861 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.642575 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.642599 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.642611 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643103 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643113 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643130 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:19:29.902505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643113 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643782 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643798 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.643806 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.644554 2585 policy_none.go:49] "None policy: Start" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.644569 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.644588 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.644651 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-142.ec2.internal.18c4c3317bc0800b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-142.ec2.internal,UID:ip-10-0-139-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-142.ec2.internal,},FirstTimestamp:2026-07-23 00:19:29.642586123 +0000 UTC m=+0.421447635,LastTimestamp:2026-07-23 00:19:29.642586123 +0000 UTC m=+0.421447635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-142.ec2.internal,}" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.645806 2585 policy_none.go:47] "Start" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.697660 2585 manager.go:337] "Starting Device Plugin manager" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.697684 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.697693 2585 server.go:87] "Starting device plugin registration server" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.697906 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.697917 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.697995 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.698079 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.698087 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.698535 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.698603 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.799005 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.800013 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.800045 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.800059 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.800089 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.806738 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.806756 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-142.ec2.internal\": node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.829637 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.922331 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.922478 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.922499 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.922536 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:29.924623 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:19:30.024080 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:29.930508 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.025603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.022920 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal"] Jul 23 00:19:30.025603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.022982 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:30.025603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.024562 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:30.025603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.024591 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:30.025603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.024604 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:30.026251 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.025786 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:30.026251 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.025885 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.026251 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.025913 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:30.026887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.026546 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:30.026887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.026547 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:30.026887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.026595 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:30.026887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.026611 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:30.026887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.026571 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:30.026887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.026643 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:30.028600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.027668 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.028600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.027701 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:19:30.028600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.028543 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:19:30.028600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.028564 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:19:30.028600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.028573 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:19:30.033787 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.030980 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.140958 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.073036 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-142.ec2.internal\" not found" node="ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.140958 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.077475 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-142.ec2.internal\" not found" node="ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.353956 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.297000 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.484408 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.397703 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.484408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.397762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6efe6a6ff7df595b5f6880a19fffd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"bd6efe6a6ff7df595b5f6880a19fffd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.484408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.397784 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6efe6a6ff7df595b5f6880a19fffd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"bd6efe6a6ff7df595b5f6880a19fffd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.484408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.397803 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42a5bd99e2334217167eb496f79450e6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-142.ec2.internal\" (UID: \"42a5bd99e2334217167eb496f79450e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.498053 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.498083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6efe6a6ff7df595b5f6880a19fffd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"bd6efe6a6ff7df595b5f6880a19fffd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.498104 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6efe6a6ff7df595b5f6880a19fffd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"bd6efe6a6ff7df595b5f6880a19fffd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.498119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42a5bd99e2334217167eb496f79450e6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-142.ec2.internal\" (UID: \"42a5bd99e2334217167eb496f79450e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.498156 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42a5bd99e2334217167eb496f79450e6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-142.ec2.internal\" (UID: \"42a5bd99e2334217167eb496f79450e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.498183 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6efe6a6ff7df595b5f6880a19fffd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"bd6efe6a6ff7df595b5f6880a19fffd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.498983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.498186 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6efe6a6ff7df595b5f6880a19fffd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"bd6efe6a6ff7df595b5f6880a19fffd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.595885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.551429 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 00:19:30.595885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.551588 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 00:19:30.603667 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:30.598203 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Jul 23 00:19:30.605531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.604371 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:19:30.605531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.604482 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.620457 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.620466 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.629246 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.630965 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.630981 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.642074 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.643140 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 00:14:31 +0000 UTC" deadline="2028-01-27 04:10:48.976240843 +0000 UTC" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.643158 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13275h51m18.333084802s" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.647134 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8r9nv" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.654634 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8r9nv" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.675420 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Jul 23 00:19:30.716995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:30.681500 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Jul 23 00:19:32.553545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:32.551283 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.080286 2585 apiserver.go:52] "Watching apiserver" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.085703 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.086114 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4z5pz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h","openshift-cluster-node-tuning-operator/tuned-7jfhp","openshift-image-registry/node-ca-j5x4l","openshift-multus/multus-additional-cni-plugins-vsxtl","openshift-multus/multus-r55pw","openshift-multus/network-metrics-daemon-4w22f","kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal","openshift-dns/node-resolver-l4m8c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal","openshift-network-diagnostics/network-check-target-gdqcl","openshift-network-operator/iptables-alerter-n85d7","openshift-ovn-kubernetes/ovnkube-node-8xlf8","kube-system/global-pull-secret-syncer-spxkd"] Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.087717 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.087776 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.089360 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.091677 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.092068 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vp6lw\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.092221 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.092401 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.092547 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.092674 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.092809 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.095426 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.096722 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.097149 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.097331 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.097454 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-krwrk\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.098511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.099380 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.099730 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.099885 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nf2lf\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100022 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100280 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100297 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100537 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 00:19:33.165765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100588 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100770 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-chzrv\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.100804 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.101529 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.102964 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.103564 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f7lv7\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.110373 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.112240 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dvbr9\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.112399 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.112827 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.113626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.115048 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.115243 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mrvtq\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.115245 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.115587 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.116063 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.116123 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.116810 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.117025 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2kst5\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.117237 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.117259 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.117605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.117659 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.118796 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.120373 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.120605 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x6dgg\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.120642 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.120661 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 00:19:33.169422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.139086 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:14:32 +0000 UTC" deadline="2028-04-17 01:58:59.919051248 +0000 UTC" Jul 23 00:19:33.170380 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.139103 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15217h39m26.779950819s" Jul 23 00:19:33.184280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184259 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-var-lib-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-cni-netd\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-hostroot\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.184373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.184495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184384 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-run-netns\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184400 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-host\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.184495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdxg\" (UniqueName: \"kubernetes.io/projected/aa64cb4e-3578-4705-addb-932930f73df9-kube-api-access-tcdxg\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.184495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-os-release\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.184495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-socket-dir-parent\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184508 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-cni-bin\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184522 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1146a88-38cb-480f-af98-ead7a0b8a6b1-tmp-dir\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkstg\" (UniqueName: \"kubernetes.io/projected/5cbd3afc-ec59-484e-ada7-00b08a723546-kube-api-access-bkstg\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184555 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-modprobe-d\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184589 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1311f755-5fa8-4691-ada1-6d855ac281c7-tmp\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-os-release\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.184665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184649 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-system-cni-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184672 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysconfig\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184695 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-slash\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-node-log\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysctl-conf\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfhc\" (UniqueName: \"kubernetes.io/projected/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-kube-api-access-9hfhc\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-systemd\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.184886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184829 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-conf-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-ovnkube-config\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cbd3afc-ec59-484e-ada7-00b08a723546-ovn-node-metrics-cert\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184953 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa64cb4e-3578-4705-addb-932930f73df9-serviceca\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.184990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185014 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-cni-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185070 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-daemon-config\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-multus-certs\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185138 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-kubelet\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185160 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-systemd\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-tuned\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.185223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185216 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185240 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brndx\" (UniqueName: \"kubernetes.io/projected/85950f20-fa7b-4aa1-9767-2dde5fd203ef-kube-api-access-brndx\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185286 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-cni-bin\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185317 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-env-overrides\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-kubernetes\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185363 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-lib-modules\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185385 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-sys\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1146a88-38cb-480f-af98-ead7a0b8a6b1-hosts-file\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-etc-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185490 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysctl-d\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185514 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-system-cni-dir\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cnibin\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185561 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.185668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-ovn\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185604 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-etc-kubernetes\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185650 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-var-lib-kubelet\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185672 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4sm\" (UniqueName: \"kubernetes.io/projected/1311f755-5fa8-4691-ada1-6d855ac281c7-kube-api-access-6h4sm\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-kubelet\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-netns\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185741 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-cni-multus\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-log-socket\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa64cb4e-3578-4705-addb-932930f73df9-host\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185823 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-cnibin\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185844 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-cni-binary-copy\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-ovnkube-script-lib\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185887 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj65p\" (UniqueName: \"kubernetes.io/projected/e1146a88-38cb-480f-af98-ead7a0b8a6b1-kube-api-access-hj65p\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.186066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185909 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-systemd-units\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.186473 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-run\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.186473 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-k8s-cni-cncf-io\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186473 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vchs\" (UniqueName: \"kubernetes.io/projected/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-kube-api-access-8vchs\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.186473 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.185991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.203135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.203119 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 00:19:33.286162 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-ovnkube-script-lib\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.286239 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj65p\" (UniqueName: \"kubernetes.io/projected/e1146a88-38cb-480f-af98-ead7a0b8a6b1-kube-api-access-hj65p\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.286239 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-systemd-units\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.286320 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-run\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286576 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-systemd-units\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286632 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-k8s-cni-cncf-io\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286684 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vchs\" (UniqueName: \"kubernetes.io/projected/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-kube-api-access-8vchs\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286704 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286768 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-var-lib-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-cni-netd\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286836 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-hostroot\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.286893 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-iptables-alerter-script\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287010 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-var-lib-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-cni-netd\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-hostroot\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-sys-fs\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.287591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-run\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.287591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-run-netns\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-host\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.287591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdxg\" (UniqueName: \"kubernetes.io/projected/aa64cb4e-3578-4705-addb-932930f73df9-kube-api-access-tcdxg\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.287750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-os-release\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287637 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-k8s-cni-cncf-io\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-socket-dir-parent\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-cni-bin\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-kubelet-config\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.287750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287741 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-socket-dir-parent\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287762 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1146a88-38cb-480f-af98-ead7a0b8a6b1-tmp-dir\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287790 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-run-netns\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkstg\" (UniqueName: \"kubernetes.io/projected/5cbd3afc-ec59-484e-ada7-00b08a723546-kube-api-access-bkstg\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-modprobe-d\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287864 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1311f755-5fa8-4691-ada1-6d855ac281c7-tmp\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-os-release\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.287940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287934 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-system-cni-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysconfig\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288035 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-slash\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-node-log\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288087 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-os-release\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288096 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysctl-conf\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfhc\" (UniqueName: \"kubernetes.io/projected/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-kube-api-access-9hfhc\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.288171 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288163 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-systemd\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288194 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd35f018-be61-4835-9c93-056a04942616-konnectivity-ca\") pod \"konnectivity-agent-4z5pz\" (UID: \"dd35f018-be61-4835-9c93-056a04942616\") " pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.288379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.288379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288257 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-conf-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288288 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-ovnkube-config\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cbd3afc-ec59-484e-ada7-00b08a723546-ovn-node-metrics-cert\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288555 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa64cb4e-3578-4705-addb-932930f73df9-serviceca\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.288555 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288458 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.288555 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-registration-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.288555 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288682 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288617 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-slash\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.288712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288697 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-cni-bin\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288743 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.287699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-host\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.288743 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-modprobe-d\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.288796 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-cni-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288842 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-daemon-config\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-multus-certs\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.288915 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288906 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-socket-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.288944 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288933 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsbl\" (UniqueName: \"kubernetes.io/projected/3de7ea09-d968-46ff-8dc2-ff27da49b770-kube-api-access-rbsbl\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.288978 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-kubelet\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289025 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288997 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-systemd\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.289075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.288999 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 00:19:33.289075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-tuned\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.289075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289068 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289221 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brndx\" (UniqueName: \"kubernetes.io/projected/85950f20-fa7b-4aa1-9767-2dde5fd203ef-kube-api-access-brndx\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289221 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289144 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-device-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.289221 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-cni-bin\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289221 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289215 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-env-overrides\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289250 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-kubernetes\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.289421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289283 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-lib-modules\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.289421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-dbus\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.289421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlzb\" (UniqueName: \"kubernetes.io/projected/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-kube-api-access-nqlzb\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.289421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289371 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-sys\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.289421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1146a88-38cb-480f-af98-ead7a0b8a6b1-hosts-file\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-etc-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289488 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289490 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysctl-d\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289610 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-node-log\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-system-cni-dir\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289662 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-system-cni-dir\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289664 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cnibin\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.289717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289700 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-ovn\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289765 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-etc-kubernetes\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysctl-d\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289800 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-host-slash\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289870 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-os-release\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-var-lib-kubelet\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4sm\" (UniqueName: \"kubernetes.io/projected/1311f755-5fa8-4691-ada1-6d855ac281c7-kube-api-access-6h4sm\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289967 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1146a88-38cb-480f-af98-ead7a0b8a6b1-tmp-dir\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.289975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-kubelet\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd35f018-be61-4835-9c93-056a04942616-agent-certs\") pod \"konnectivity-agent-4z5pz\" (UID: \"dd35f018-be61-4835-9c93-056a04942616\") " pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290035 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-netns\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290066 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-cni-multus\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290202 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.290093 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-log-socket\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa64cb4e-3578-4705-addb-932930f73df9-host\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.290162 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:33.790144315 +0000 UTC m=+3.086839280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-cnibin\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290193 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa64cb4e-3578-4705-addb-932930f73df9-host\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-cni-binary-copy\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290366 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290654 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-ovnkube-script-lib\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290752 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-multus-certs\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290810 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-cni-binary-copy\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.290935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290845 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.291436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysctl-conf\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.291436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.290970 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-cni-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.291436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.291220 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-systemd\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.291436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.291310 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-conf-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.291644 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.291481 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-multus-daemon-config\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.291699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.291681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa64cb4e-3578-4705-addb-932930f73df9-serviceca\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.291986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.291963 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-cnibin\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.292130 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-system-cni-dir\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292649 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-cni-bin\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-lib-modules\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292703 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-cni-multus\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292710 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-sysconfig\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292749 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1146a88-38cb-480f-af98-ead7a0b8a6b1-hosts-file\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292802 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-etc-openvswitch\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292806 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-ovnkube-config\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292833 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-tuned\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.292869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292842 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cnibin\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.293258 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292928 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.293258 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-var-lib-kubelet\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.293258 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293188 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-etc-kubernetes\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.293258 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.292743 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1311f755-5fa8-4691-ada1-6d855ac281c7-tmp\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.293258 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293239 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-var-lib-kubelet\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.293427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293284 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-host-kubelet\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.293427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-run-ovn\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.293427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-systemd\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.293427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293419 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85950f20-fa7b-4aa1-9767-2dde5fd203ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.293608 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-host-run-netns\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.293608 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293534 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-etc-kubernetes\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.293608 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293560 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5cbd3afc-ec59-484e-ada7-00b08a723546-log-socket\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.293811 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.293624 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1311f755-5fa8-4691-ada1-6d855ac281c7-sys\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.294692 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.294648 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5cbd3afc-ec59-484e-ada7-00b08a723546-env-overrides\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.294990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.294969 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cbd3afc-ec59-484e-ada7-00b08a723546-ovn-node-metrics-cert\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.296058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.296026 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj65p\" (UniqueName: \"kubernetes.io/projected/e1146a88-38cb-480f-af98-ead7a0b8a6b1-kube-api-access-hj65p\") pod \"node-resolver-l4m8c\" (UID: \"e1146a88-38cb-480f-af98-ead7a0b8a6b1\") " pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.296543 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.296525 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdxg\" (UniqueName: \"kubernetes.io/projected/aa64cb4e-3578-4705-addb-932930f73df9-kube-api-access-tcdxg\") pod \"node-ca-j5x4l\" (UID: \"aa64cb4e-3578-4705-addb-932930f73df9\") " pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.296617 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.296577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vchs\" (UniqueName: \"kubernetes.io/projected/f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b-kube-api-access-8vchs\") pod \"multus-r55pw\" (UID: \"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b\") " pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.297885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.297861 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brndx\" (UniqueName: \"kubernetes.io/projected/85950f20-fa7b-4aa1-9767-2dde5fd203ef-kube-api-access-brndx\") pod \"multus-additional-cni-plugins-vsxtl\" (UID: \"85950f20-fa7b-4aa1-9767-2dde5fd203ef\") " pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.298949 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.298920 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfhc\" (UniqueName: \"kubernetes.io/projected/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-kube-api-access-9hfhc\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.299582 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.299555 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4sm\" (UniqueName: \"kubernetes.io/projected/1311f755-5fa8-4691-ada1-6d855ac281c7-kube-api-access-6h4sm\") pod \"tuned-7jfhp\" (UID: \"1311f755-5fa8-4691-ada1-6d855ac281c7\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.299629 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.299585 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkstg\" (UniqueName: \"kubernetes.io/projected/5cbd3afc-ec59-484e-ada7-00b08a723546-kube-api-access-bkstg\") pod \"ovnkube-node-8xlf8\" (UID: \"5cbd3afc-ec59-484e-ada7-00b08a723546\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.309798 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.309782 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l4m8c" Jul 23 00:19:33.391223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391197 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.391335 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-host-slash\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.391335 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391298 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd35f018-be61-4835-9c93-056a04942616-agent-certs\") pod \"konnectivity-agent-4z5pz\" (UID: \"dd35f018-be61-4835-9c93-056a04942616\") " pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.391420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391331 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-host-slash\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.391420 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.391340 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:33.391420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391380 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:33.391420 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.391416 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:19:33.891395701 +0000 UTC m=+3.188090659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:33.391635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391452 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-iptables-alerter-script\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.391635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391499 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-sys-fs\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.391635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391543 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-kubelet-config\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.391635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.391834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd35f018-be61-4835-9c93-056a04942616-konnectivity-ca\") pod \"konnectivity-agent-4z5pz\" (UID: \"dd35f018-be61-4835-9c93-056a04942616\") " pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.391834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.391834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-registration-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.391960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391807 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-kubelet-config\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.391960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391944 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392055 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.391992 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-registration-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392102 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392067 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-iptables-alerter-script\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.392102 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392071 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392127 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-sys-fs\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392139 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-socket-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392346 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392233 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd35f018-be61-4835-9c93-056a04942616-konnectivity-ca\") pod \"konnectivity-agent-4z5pz\" (UID: \"dd35f018-be61-4835-9c93-056a04942616\") " pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.392402 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392350 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsbl\" (UniqueName: \"kubernetes.io/projected/3de7ea09-d968-46ff-8dc2-ff27da49b770-kube-api-access-rbsbl\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-device-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392518 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392452 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-socket-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392518 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392500 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-dbus\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.392620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlzb\" (UniqueName: \"kubernetes.io/projected/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-kube-api-access-nqlzb\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.392620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392535 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3de7ea09-d968-46ff-8dc2-ff27da49b770-device-dir\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.392707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.392640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-dbus\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.393813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.393794 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd35f018-be61-4835-9c93-056a04942616-agent-certs\") pod \"konnectivity-agent-4z5pz\" (UID: \"dd35f018-be61-4835-9c93-056a04942616\") " pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.394396 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.394381 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:19:33.397867 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.397851 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:19:33.397867 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.397867 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:19:33.397957 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.397877 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:33.397957 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.397930 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:33.897918653 +0000 UTC m=+3.194613598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:33.400460 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.400440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlzb\" (UniqueName: \"kubernetes.io/projected/c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae-kube-api-access-nqlzb\") pod \"iptables-alerter-n85d7\" (UID: \"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae\") " pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.400540 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.400444 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsbl\" (UniqueName: \"kubernetes.io/projected/3de7ea09-d968-46ff-8dc2-ff27da49b770-kube-api-access-rbsbl\") pod \"aws-ebs-csi-driver-node-l9r5h\" (UID: \"3de7ea09-d968-46ff-8dc2-ff27da49b770\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.520076 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.519999 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:19:33.541591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.541569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" Jul 23 00:19:33.547092 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.547076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j5x4l" Jul 23 00:19:33.577556 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.577538 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" Jul 23 00:19:33.592218 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.592202 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r55pw" Jul 23 00:19:33.623924 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.623900 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:33.638747 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.638726 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" Jul 23 00:19:33.644732 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.644718 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n85d7" Jul 23 00:19:33.769519 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:33.769488 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1146a88_38cb_480f_af98_ead7a0b8a6b1.slice/crio-1de446dccbaf776440e3c6f36caa7e44d7ba1402510e348987ace71cf6641fc5 WatchSource:0}: Error finding container 1de446dccbaf776440e3c6f36caa7e44d7ba1402510e348987ace71cf6641fc5: Status 404 returned error can't find the container with id 1de446dccbaf776440e3c6f36caa7e44d7ba1402510e348987ace71cf6641fc5 Jul 23 00:19:33.769730 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:33.769709 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a5bd99e2334217167eb496f79450e6.slice/crio-578a9a7706680214bab47591e25ac931aa7cb579a3d749f715e3ac9a390f3c37 WatchSource:0}: Error finding container 578a9a7706680214bab47591e25ac931aa7cb579a3d749f715e3ac9a390f3c37: Status 404 returned error can't find the container with id 578a9a7706680214bab47591e25ac931aa7cb579a3d749f715e3ac9a390f3c37 Jul 23 00:19:33.769918 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:33.769893 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6efe6a6ff7df595b5f6880a19fffd7.slice/crio-bf48996e161d26effdd37eb60d6bbd0fbe6229245f39cd1cc5c964b35f344197 WatchSource:0}: Error finding container bf48996e161d26effdd37eb60d6bbd0fbe6229245f39cd1cc5c964b35f344197: Status 404 returned error can't find the container with id bf48996e161d26effdd37eb60d6bbd0fbe6229245f39cd1cc5c964b35f344197 Jul 23 00:19:33.773817 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.773801 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:19:33.795214 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.795193 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:33.795325 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.795309 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:33.795372 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.795364 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:34.795346878 +0000 UTC m=+4.092041827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:33.807963 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:33.807939 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbd3afc_ec59_484e_ada7_00b08a723546.slice/crio-0afa1b89cfbea1806a575c6067d0c35ce4d2c7f271f106f9e4a3e704ca1b7f9f WatchSource:0}: Error finding container 0afa1b89cfbea1806a575c6067d0c35ce4d2c7f271f106f9e4a3e704ca1b7f9f: Status 404 returned error can't find the container with id 0afa1b89cfbea1806a575c6067d0c35ce4d2c7f271f106f9e4a3e704ca1b7f9f Jul 23 00:19:33.896373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.896336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:33.896541 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.896448 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:33.896541 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.896516 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:19:34.896498344 +0000 UTC m=+4.193193292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:33.900881 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:33.900863 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85950f20_fa7b_4aa1_9767_2dde5fd203ef.slice/crio-6453c4a3e5ac0ec503fd7eab35844c1673c2238a0c78bd9c5abbb8d16ac953d5 WatchSource:0}: Error finding container 6453c4a3e5ac0ec503fd7eab35844c1673c2238a0c78bd9c5abbb8d16ac953d5: Status 404 returned error can't find the container with id 6453c4a3e5ac0ec503fd7eab35844c1673c2238a0c78bd9c5abbb8d16ac953d5 Jul 23 00:19:33.965964 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:33.965936 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1efbed5_7c8e_44bf_8f15_ac8cd1dc205b.slice/crio-0efd9703ff4d5aecde8a9e89efea7e77b40498ac6812738a26f9fdeabf21d7c5 WatchSource:0}: Error finding container 0efd9703ff4d5aecde8a9e89efea7e77b40498ac6812738a26f9fdeabf21d7c5: Status 404 returned error can't find the container with id 0efd9703ff4d5aecde8a9e89efea7e77b40498ac6812738a26f9fdeabf21d7c5 Jul 23 00:19:33.997313 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:33.997286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:33.997458 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.997441 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:19:33.997458 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.997472 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:19:33.997563 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.997482 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:33.997563 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:33.997525 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:34.997511121 +0000 UTC m=+4.294206067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:34.000752 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:34.000729 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5020952_35cc_4c1a_bb0c_b02ad1e0b0ae.slice/crio-592cd16e6d04b03f8e94e459f064a52a2d946e19db118bf689398dc9d7630a84 WatchSource:0}: Error finding container 592cd16e6d04b03f8e94e459f064a52a2d946e19db118bf689398dc9d7630a84: Status 404 returned error can't find the container with id 592cd16e6d04b03f8e94e459f064a52a2d946e19db118bf689398dc9d7630a84 Jul 23 00:19:34.056479 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:34.056444 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd35f018_be61_4835_9c93_056a04942616.slice/crio-c486655cdb693b9b9d7995aff0a3182ae4a821b46071c74fa744414fd0e5bcac WatchSource:0}: Error finding container c486655cdb693b9b9d7995aff0a3182ae4a821b46071c74fa744414fd0e5bcac: Status 404 returned error can't find the container with id c486655cdb693b9b9d7995aff0a3182ae4a821b46071c74fa744414fd0e5bcac Jul 23 00:19:34.075708 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:34.075677 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de7ea09_d968_46ff_8dc2_ff27da49b770.slice/crio-818ecdab7acbcee6a645db77ea29d6f3828cc1e8e4deafcdd9420fa9b9747549 WatchSource:0}: Error finding container 818ecdab7acbcee6a645db77ea29d6f3828cc1e8e4deafcdd9420fa9b9747549: Status 404 returned error can't find the container with id 818ecdab7acbcee6a645db77ea29d6f3828cc1e8e4deafcdd9420fa9b9747549 Jul 23 00:19:34.115316 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:34.115287 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1311f755_5fa8_4691_ada1_6d855ac281c7.slice/crio-28ce67c4eb6ff3c04bc04e1528437f3525bcfb917b12e90c85a9b1ad37e120af WatchSource:0}: Error finding container 28ce67c4eb6ff3c04bc04e1528437f3525bcfb917b12e90c85a9b1ad37e120af: Status 404 returned error can't find the container with id 28ce67c4eb6ff3c04bc04e1528437f3525bcfb917b12e90c85a9b1ad37e120af Jul 23 00:19:34.120549 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.120510 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" event={"ID":"1311f755-5fa8-4691-ada1-6d855ac281c7","Type":"ContainerStarted","Data":"28ce67c4eb6ff3c04bc04e1528437f3525bcfb917b12e90c85a9b1ad37e120af"} Jul 23 00:19:34.121492 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.121451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4z5pz" event={"ID":"dd35f018-be61-4835-9c93-056a04942616","Type":"ContainerStarted","Data":"c486655cdb693b9b9d7995aff0a3182ae4a821b46071c74fa744414fd0e5bcac"} Jul 23 00:19:34.122516 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.122497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r55pw" event={"ID":"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b","Type":"ContainerStarted","Data":"0efd9703ff4d5aecde8a9e89efea7e77b40498ac6812738a26f9fdeabf21d7c5"} Jul 23 00:19:34.123345 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.123326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerStarted","Data":"6453c4a3e5ac0ec503fd7eab35844c1673c2238a0c78bd9c5abbb8d16ac953d5"} Jul 23 00:19:34.124185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.124160 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" event={"ID":"bd6efe6a6ff7df595b5f6880a19fffd7","Type":"ContainerStarted","Data":"bf48996e161d26effdd37eb60d6bbd0fbe6229245f39cd1cc5c964b35f344197"} Jul 23 00:19:34.125034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.125017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l4m8c" event={"ID":"e1146a88-38cb-480f-af98-ead7a0b8a6b1","Type":"ContainerStarted","Data":"1de446dccbaf776440e3c6f36caa7e44d7ba1402510e348987ace71cf6641fc5"} Jul 23 00:19:34.125865 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.125849 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" event={"ID":"3de7ea09-d968-46ff-8dc2-ff27da49b770","Type":"ContainerStarted","Data":"818ecdab7acbcee6a645db77ea29d6f3828cc1e8e4deafcdd9420fa9b9747549"} Jul 23 00:19:34.126668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.126653 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n85d7" event={"ID":"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae","Type":"ContainerStarted","Data":"592cd16e6d04b03f8e94e459f064a52a2d946e19db118bf689398dc9d7630a84"} Jul 23 00:19:34.127475 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.127446 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"0afa1b89cfbea1806a575c6067d0c35ce4d2c7f271f106f9e4a3e704ca1b7f9f"} Jul 23 00:19:34.128403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.128385 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" event={"ID":"42a5bd99e2334217167eb496f79450e6","Type":"ContainerStarted","Data":"578a9a7706680214bab47591e25ac931aa7cb579a3d749f715e3ac9a390f3c37"} Jul 23 00:19:34.139243 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.139206 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:14:32 +0000 UTC" deadline="2028-02-25 06:49:38.34700861 +0000 UTC" Jul 23 00:19:34.139243 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.139242 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13974h30m4.207768972s" Jul 23 00:19:34.189391 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:19:34.189374 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa64cb4e_3578_4705_addb_932930f73df9.slice/crio-8c126d79d05dd28d5de5dad16138cdafc66d663dd588ca1e7300dd6acbdebd83 WatchSource:0}: Error finding container 8c126d79d05dd28d5de5dad16138cdafc66d663dd588ca1e7300dd6acbdebd83: Status 404 returned error can't find the container with id 8c126d79d05dd28d5de5dad16138cdafc66d663dd588ca1e7300dd6acbdebd83 Jul 23 00:19:34.405263 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.405190 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:34.405417 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:34.405351 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:34.405772 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.405753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:34.405885 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:34.405865 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:34.803474 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.803343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:34.803629 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:34.803521 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:34.803629 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:34.803584 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:36.803565664 +0000 UTC m=+6.100260613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:34.904400 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:34.904359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:34.904589 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:34.904553 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:34.904651 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:34.904614 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:19:36.90459701 +0000 UTC m=+6.201291957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:35.005312 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:35.005277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:35.005520 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:35.005449 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:19:35.005520 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:35.005484 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:19:35.005520 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:35.005498 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:35.005687 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:35.005557 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:37.005537067 +0000 UTC m=+6.302232019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:35.136627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:35.136510 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j5x4l" event={"ID":"aa64cb4e-3578-4705-addb-932930f73df9","Type":"ContainerStarted","Data":"8c126d79d05dd28d5de5dad16138cdafc66d663dd588ca1e7300dd6acbdebd83"} Jul 23 00:19:35.408453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:35.407100 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:35.408453 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:35.407224 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:36.406257 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:36.405594 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:36.406257 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:36.405736 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:36.406257 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:36.406106 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:36.406257 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:36.406195 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:36.821053 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:36.820968 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:36.821743 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:36.821115 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:36.821743 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:36.821179 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:40.821159124 +0000 UTC m=+10.117854071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:36.921553 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:36.921515 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:36.921730 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:36.921657 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:36.921730 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:36.921717 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:19:40.921698187 +0000 UTC m=+10.218393132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:37.022084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:37.022048 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:37.022284 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:37.022265 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:19:37.022365 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:37.022293 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:19:37.022365 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:37.022306 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:37.022365 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:37.022360 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:41.022342765 +0000 UTC m=+10.319037717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:37.407431 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:37.407396 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:37.407630 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:37.407549 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:38.405211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:38.405178 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:38.405725 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:38.405310 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:38.405725 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:38.405324 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:38.405725 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:38.405452 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:39.405319 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:39.405287 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:39.405730 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:39.405404 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:40.405408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:40.405134 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:40.405408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:40.405172 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:40.405408 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:40.405261 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:40.405408 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:40.405410 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:40.853370 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:40.853289 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:40.853568 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:40.853428 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:40.853568 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:40.853519 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:48.853498401 +0000 UTC m=+18.150193372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:40.953817 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:40.953778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:40.954006 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:40.953922 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:40.954006 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:40.953982 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:19:48.953965758 +0000 UTC m=+18.250660718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:41.054310 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:41.054276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:41.054521 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:41.054431 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:19:41.054521 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:41.054449 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:19:41.054521 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:41.054477 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:41.054691 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:41.054543 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:19:49.054515717 +0000 UTC m=+18.351210680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:41.406779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:41.406744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:41.407148 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:41.406853 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:42.405642 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:42.405248 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:42.405642 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:42.405267 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:42.405642 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:42.405375 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:42.405642 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:42.405425 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:43.405906 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:43.405818 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:43.406341 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:43.405941 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:44.405170 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:44.405127 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:44.405338 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:44.405244 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:44.405338 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:44.405313 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:44.405444 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:44.405417 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:45.405097 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:45.405017 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:45.405478 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:45.405159 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:46.405863 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:46.405829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:46.405863 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:46.405853 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:46.406300 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:46.405957 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:46.406300 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:46.406119 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:47.405040 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:47.405007 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:47.405214 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:47.405148 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:48.405185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:48.405133 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:48.405611 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:48.405155 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:48.405611 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:48.405293 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:48.405611 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:48.405345 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:48.913480 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:48.913438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:48.913646 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:48.913587 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:48.913688 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:48.913651 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:04.913635296 +0000 UTC m=+34.210330256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:19:49.014636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:49.014597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:49.014812 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.014734 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:49.014812 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.014803 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:20:05.014787962 +0000 UTC m=+34.311482919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:19:49.115285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:49.115245 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:49.115449 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.115386 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:19:49.115449 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.115410 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:19:49.115449 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.115423 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:49.115595 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.115494 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:05.115479284 +0000 UTC m=+34.412174247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:19:49.405721 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:49.405634 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:49.406146 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:49.405782 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:50.405567 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:50.405488 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:50.405567 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:50.405507 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:50.405785 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:50.405604 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:50.405785 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:50.405733 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:51.406259 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:51.406224 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:51.406717 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:51.406347 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:52.405596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:52.405561 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:52.405783 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:52.405677 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:52.405783 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:52.405736 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:52.405901 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:52.405840 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:53.405876 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:53.405845 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:53.406328 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:53.405971 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:54.405765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:54.405447 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:54.405920 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:54.405653 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:54.405920 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:54.405885 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:54.406302 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:54.405916 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:55.182809 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.182782 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" event={"ID":"1311f755-5fa8-4691-ada1-6d855ac281c7","Type":"ContainerStarted","Data":"1a1f9d8dec4d3bd591bb1eff0c31f683f56a355e9b11fc5e01aff5d5aca667e0"} Jul 23 00:19:55.184141 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.184113 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4z5pz" event={"ID":"dd35f018-be61-4835-9c93-056a04942616","Type":"ContainerStarted","Data":"512a9f91a59d418bae56ef27391999ba1b2ded0dfc7dc758256929c856c7b40a"} Jul 23 00:19:55.185368 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.185336 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r55pw" event={"ID":"f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b","Type":"ContainerStarted","Data":"51ffd94204f3a53564c1f348e82ed7a2903044a6ea9de03fb3651d0db50ddc7b"} Jul 23 00:19:55.186700 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.186678 2585 generic.go:356] "Generic (PLEG): container finished" podID="85950f20-fa7b-4aa1-9767-2dde5fd203ef" containerID="dfaf40833e7a718d6635080e4acbd80f9a3c122590d84fa42a891d2ab0c22d8b" exitCode=0 Jul 23 00:19:55.186790 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.186757 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerDied","Data":"dfaf40833e7a718d6635080e4acbd80f9a3c122590d84fa42a891d2ab0c22d8b"} Jul 23 00:19:55.188153 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.188136 2585 generic.go:356] "Generic (PLEG): container finished" podID="bd6efe6a6ff7df595b5f6880a19fffd7" containerID="bb8e46e78150afd6aa43a16ff6f4ff20976c6e141b479133c95d3885a2c8b3c0" exitCode=0 Jul 23 00:19:55.188227 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.188200 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" event={"ID":"bd6efe6a6ff7df595b5f6880a19fffd7","Type":"ContainerDied","Data":"bb8e46e78150afd6aa43a16ff6f4ff20976c6e141b479133c95d3885a2c8b3c0"} Jul 23 00:19:55.189713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.189688 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l4m8c" event={"ID":"e1146a88-38cb-480f-af98-ead7a0b8a6b1","Type":"ContainerStarted","Data":"827aa1a43e98912d74a24f9c9941685893dee57a033db22af2b6e912101d234e"} Jul 23 00:19:55.191077 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.191055 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" event={"ID":"3de7ea09-d968-46ff-8dc2-ff27da49b770","Type":"ContainerStarted","Data":"d3df358b952ebdc9a8502bb9fb5f421b29dbbe8cea40c0c3021836c73ebbdfdd"} Jul 23 00:19:55.193986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.193964 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"ffa4514a18c33286980c8388df6242ac6b2826e66f74f1f787d4498ae4505ebf"} Jul 23 00:19:55.194062 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.194038 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"cb56524e30326c6d7697f450de668ccb3f856005e7821f66f314b3c9c5b04fa3"} Jul 23 00:19:55.194106 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.194086 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"d363e0ce9d36ff8ec42c7127cadbe27b4c1c25846096dc08eeff123d5c05f7f1"} Jul 23 00:19:55.194150 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.194134 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"9a5f7329ba67dfbf091b0dd07f34756c9baae4ddfb09683070e1358f3e35ad4a"} Jul 23 00:19:55.194200 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.194179 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"6c7daba2a53d8e7ac77b1eb81f02258b25eb6502b2e402292d8c8d2d0c50342d"} Jul 23 00:19:55.194252 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.194227 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"d8d905b51d0e9bdbe249d2f3f0dfade16f31577cbd6474916023161c31d3438d"} Jul 23 00:19:55.195296 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.195274 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" event={"ID":"42a5bd99e2334217167eb496f79450e6","Type":"ContainerStarted","Data":"70ae034f1b3000125b409481e31f675681684feaa064139883022210d1e98df0"} Jul 23 00:19:55.196564 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.196545 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j5x4l" event={"ID":"aa64cb4e-3578-4705-addb-932930f73df9","Type":"ContainerStarted","Data":"717f022d9a416d5184d769cfb3c522c5b919e136613a3b2acb8efc66bab1b298"} Jul 23 00:19:55.198599 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.198563 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7jfhp" podStartSLOduration=4.263635954 podStartE2EDuration="24.198553402s" totalImagesPullingTime="19.934917448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:55.198552375 +0000 UTC m=+24.495247344" watchObservedRunningTime="2026-07-23 00:19:55.198553402 +0000 UTC m=+24.495248366" Jul 23 00:19:55.257449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.257208 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r55pw" podStartSLOduration=4.137983546 podStartE2EDuration="24.257190086s" totalImagesPullingTime="20.11920654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:55.256652899 +0000 UTC m=+24.553347895" watchObservedRunningTime="2026-07-23 00:19:55.257190086 +0000 UTC m=+24.553885055" Jul 23 00:19:55.269529 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.269484 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l4m8c" podStartSLOduration=3.991549596 podStartE2EDuration="24.269443975s" totalImagesPullingTime="20.277894379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:55.269325847 +0000 UTC m=+24.566020840" watchObservedRunningTime="2026-07-23 00:19:55.269443975 +0000 UTC m=+24.566138944" Jul 23 00:19:55.294736 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.294701 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" podStartSLOduration=23.294688646 podStartE2EDuration="23.294688646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:55.294372947 +0000 UTC m=+24.591067935" watchObservedRunningTime="2026-07-23 00:19:55.294688646 +0000 UTC m=+24.591383613" Jul 23 00:19:55.295096 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.295071 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4z5pz" podStartSLOduration=8.332080792 podStartE2EDuration="24.295063198s" totalImagesPullingTime="15.962982406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:55.282391325 +0000 UTC m=+24.579086295" watchObservedRunningTime="2026-07-23 00:19:55.295063198 +0000 UTC m=+24.591758165" Jul 23 00:19:55.308458 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.308418 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j5x4l" podStartSLOduration=4.449732484 podStartE2EDuration="24.308404766s" totalImagesPullingTime="19.858672282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:55.308350574 +0000 UTC m=+24.605045543" watchObservedRunningTime="2026-07-23 00:19:55.308404766 +0000 UTC m=+24.605099735" Jul 23 00:19:55.405014 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.404985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:55.405206 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:55.405129 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:55.986696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:55.986675 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 00:19:56.192960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.192840 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T00:19:55.986692759Z","UUID":"b76ec7cb-0d13-4429-a5d3-dc6de08f66be","Handler":null,"Name":"","Endpoint":""} Jul 23 00:19:56.194321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.194297 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 00:19:56.194436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.194339 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 00:19:56.200599 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.200573 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" event={"ID":"3de7ea09-d968-46ff-8dc2-ff27da49b770","Type":"ContainerStarted","Data":"c319ed539416e7f80d545686a1072edf65a6f11dabe414b83842f66228772795"} Jul 23 00:19:56.202270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.202244 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n85d7" event={"ID":"c5020952-35cc-4c1a-bb0c-b02ad1e0b0ae","Type":"ContainerStarted","Data":"3291d07ead012d851dd3eba7cada7b53c9b1b732a2b80dd1680ce29a4720ada6"} Jul 23 00:19:56.204117 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.204058 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" event={"ID":"bd6efe6a6ff7df595b5f6880a19fffd7","Type":"ContainerStarted","Data":"efb217960efb445d863e39dbef7418ecbfc69b74532f0bafbc232cb4cf7afca2"} Jul 23 00:19:56.221157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.221120 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n85d7" podStartSLOduration=9.20228304 podStartE2EDuration="25.221103451s" totalImagesPullingTime="16.018820411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:56.220562089 +0000 UTC m=+25.517257106" watchObservedRunningTime="2026-07-23 00:19:56.221103451 +0000 UTC m=+25.517798419" Jul 23 00:19:56.232759 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.232725 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" podStartSLOduration=24.23271663 podStartE2EDuration="24.23271663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:56.232044795 +0000 UTC m=+25.528739763" watchObservedRunningTime="2026-07-23 00:19:56.23271663 +0000 UTC m=+25.529411598" Jul 23 00:19:56.405197 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.405163 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:56.405447 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:56.405300 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:56.405546 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:56.405495 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:56.405637 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:56.405611 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:57.211227 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:57.211198 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"697cd1078bbfa3fc537a51c03cb63f4de814439166f4b7e5ff665586293f1d8a"} Jul 23 00:19:57.405637 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:57.405597 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:57.405797 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:57.405725 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:19:58.215212 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:58.215177 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" event={"ID":"3de7ea09-d968-46ff-8dc2-ff27da49b770","Type":"ContainerStarted","Data":"fed9fed6526dd55b8a12d57625abafd3479b5fb3367315ca28ec52634d2b4a7c"} Jul 23 00:19:58.230640 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:58.230597 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9r5h" podStartSLOduration=4.515756022 podStartE2EDuration="27.230584713s" totalImagesPullingTime="22.714828691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:58.23043171 +0000 UTC m=+27.527126688" watchObservedRunningTime="2026-07-23 00:19:58.230584713 +0000 UTC m=+27.527279681" Jul 23 00:19:58.338693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:58.338655 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:58.339339 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:58.339316 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:58.405713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:58.405679 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:19:58.405880 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:58.405678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:19:58.405880 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:58.405768 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:19:58.405976 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:58.405893 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:19:59.218288 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:59.218222 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:59.218760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:59.218735 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4z5pz" Jul 23 00:19:59.405076 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:19:59.405038 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:19:59.405224 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:19:59.405192 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:20:00.223906 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:00.223754 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" event={"ID":"5cbd3afc-ec59-484e-ada7-00b08a723546","Type":"ContainerStarted","Data":"0183d49a10caf0a313bc3b0be80f34fda0f84258c302d6237c9a4b92663d2c36"} Jul 23 00:20:00.224503 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:00.224221 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:20:00.238954 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:00.238937 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:20:00.248272 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:00.248234 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" podStartSLOduration=8.803338312 podStartE2EDuration="29.248222421s" totalImagesPullingTime="20.444884109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:00.2466553 +0000 UTC m=+29.543350305" watchObservedRunningTime="2026-07-23 00:20:00.248222421 +0000 UTC m=+29.544917388" Jul 23 00:20:00.405654 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:00.405592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:00.405760 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:00.405693 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:20:00.405821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:00.405758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:00.405890 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:00.405865 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:20:01.227592 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:01.227563 2585 generic.go:356] "Generic (PLEG): container finished" podID="85950f20-fa7b-4aa1-9767-2dde5fd203ef" containerID="1b762dc7102221810179b784fbbe2808059313ea2de4ce6825bfd52b9d137db4" exitCode=0 Jul 23 00:20:01.228159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:01.227650 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerDied","Data":"1b762dc7102221810179b784fbbe2808059313ea2de4ce6825bfd52b9d137db4"} Jul 23 00:20:01.228159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:01.228058 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:20:01.228159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:01.228134 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:20:01.242745 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:01.242726 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:20:01.405705 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:01.405678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:01.405845 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:01.405766 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:20:02.111766 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:02.111533 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-spxkd"] Jul 23 00:20:02.114292 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:02.114266 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4w22f"] Jul 23 00:20:02.114443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:02.114403 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:02.114575 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:02.114548 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:20:02.125589 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:02.125527 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gdqcl"] Jul 23 00:20:02.125718 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:02.125625 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:02.125718 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:02.125707 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:20:02.229633 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:02.229602 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:02.230047 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:02.229719 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:20:03.234846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:03.234759 2585 generic.go:356] "Generic (PLEG): container finished" podID="85950f20-fa7b-4aa1-9767-2dde5fd203ef" containerID="f440a1ffde36b25cf21838563e1075afc3be81cad6d6e3ade1317bbb4e6f9cb0" exitCode=0 Jul 23 00:20:03.235229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:03.234839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerDied","Data":"f440a1ffde36b25cf21838563e1075afc3be81cad6d6e3ade1317bbb4e6f9cb0"} Jul 23 00:20:04.404975 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:04.404948 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:04.405312 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:04.405038 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:04.405312 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:04.405144 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:20:04.405312 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:04.405039 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:20:04.405312 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:04.405044 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:04.405312 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:04.405224 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:20:04.936098 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:04.936064 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:04.936275 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:04.936171 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:20:04.936275 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:04.936227 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs podName:ac42cc77-e8f3-4ee1-a85b-03b0750efc61 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:36.936211129 +0000 UTC m=+66.232906080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs") pod "network-metrics-daemon-4w22f" (UID: "ac42cc77-e8f3-4ee1-a85b-03b0750efc61") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:20:05.036689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:05.036654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:05.036853 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.036793 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:20:05.036896 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.036855 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret podName:cd3be28b-aa8d-4a5a-b230-dbedffd5543c nodeName:}" failed. No retries permitted until 2026-07-23 00:20:37.036837254 +0000 UTC m=+66.333532202 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret") pod "global-pull-secret-syncer-spxkd" (UID: "cd3be28b-aa8d-4a5a-b230-dbedffd5543c") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:20:05.137720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:05.137691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:05.137866 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.137795 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:20:05.137866 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.137810 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:20:05.137866 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.137818 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nqprx for pod openshift-network-diagnostics/network-check-target-gdqcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:20:05.137866 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.137859 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx podName:e04e9d9a-ede9-4fa6-8857-e78e80726373 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:37.137846484 +0000 UTC m=+66.434541431 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqprx" (UniqueName: "kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx") pod "network-check-target-gdqcl" (UID: "e04e9d9a-ede9-4fa6-8857-e78e80726373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:20:05.241263 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:05.241190 2585 generic.go:356] "Generic (PLEG): container finished" podID="85950f20-fa7b-4aa1-9767-2dde5fd203ef" containerID="da798a2310d467e38cacc09bb14b2dbd82c702e49a88b9154d1e74d95943e15c" exitCode=0 Jul 23 00:20:05.241263 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:05.241247 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerDied","Data":"da798a2310d467e38cacc09bb14b2dbd82c702e49a88b9154d1e74d95943e15c"} Jul 23 00:20:05.407818 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:05.407791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:05.408172 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:05.407885 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:20:06.405475 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:06.405445 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:06.405661 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:06.405445 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:06.405661 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:06.405577 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqcl" podUID="e04e9d9a-ede9-4fa6-8857-e78e80726373" Jul 23 00:20:06.405661 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:06.405647 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxkd" podUID="cd3be28b-aa8d-4a5a-b230-dbedffd5543c" Jul 23 00:20:07.405035 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:07.405006 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:07.405480 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:07.405160 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4w22f" podUID="ac42cc77-e8f3-4ee1-a85b-03b0750efc61" Jul 23 00:20:08.020343 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.020273 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeReady" Jul 23 00:20:08.020507 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.020422 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 00:20:08.054032 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.054006 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4"] Jul 23 00:20:08.055928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.055909 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.058212 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.058184 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 00:20:08.058931 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.058405 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f665b65c-w5xjk"] Jul 23 00:20:08.058931 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.058668 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t6tl2\"" Jul 23 00:20:08.058931 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.058888 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 00:20:08.060491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.060454 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp"] Jul 23 00:20:08.060600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.060558 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.064058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.063149 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 00:20:08.064058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.063303 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw"] Jul 23 00:20:08.064058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.063592 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 00:20:08.064058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.063750 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 00:20:08.064058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.063849 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8wqt5\"" Jul 23 00:20:08.064360 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.064165 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.065663 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.065639 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8kf92"] Jul 23 00:20:08.065748 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.065700 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" Jul 23 00:20:08.066664 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.066643 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.067570 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.067554 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc"] Jul 23 00:20:08.067768 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.067753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.068592 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.068558 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.068592 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.068574 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 00:20:08.069055 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.069036 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.069199 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.069172 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xm7fw\"" Jul 23 00:20:08.069388 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.069368 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zhk5t\"" Jul 23 00:20:08.069558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.069435 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.070891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.070229 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 00:20:08.070891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.070512 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.070891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.070574 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.071089 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.070987 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d9qrq\"" Jul 23 00:20:08.071591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.071453 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 00:20:08.071591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.071479 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66c98f88f4-c2vjt"] Jul 23 00:20:08.071744 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.071715 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.073057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.072958 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 00:20:08.074546 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.074527 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.074822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.074800 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh"] Jul 23 00:20:08.083506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.083483 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.083872 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.083851 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 00:20:08.085700 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.085683 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb"] Jul 23 00:20:08.086713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.086672 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.087285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.087267 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.092754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.092734 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f6fc97d76-kshb5"] Jul 23 00:20:08.093728 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.093708 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.095151 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.094592 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wvjzl"] Jul 23 00:20:08.095253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.095184 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4ls9t\"" Jul 23 00:20:08.095313 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.095268 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 00:20:08.095598 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.095582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.096007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.095983 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 00:20:08.097973 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.096885 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl"] Jul 23 00:20:08.097973 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.097655 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.098262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.098235 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d68lc\"" Jul 23 00:20:08.098580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.098559 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.099246 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.098877 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 00:20:08.099246 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.099139 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 00:20:08.099383 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.099352 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.100647 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.099654 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr"] Jul 23 00:20:08.100647 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.100238 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.100647 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.100436 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jdvh7\"" Jul 23 00:20:08.100858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.100675 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 00:20:08.101104 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101072 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.101192 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101145 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 00:20:08.101352 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101334 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 00:20:08.101352 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101346 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.101572 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101543 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.101662 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101606 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wf27l\"" Jul 23 00:20:08.102815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101792 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 00:20:08.102815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f"] Jul 23 00:20:08.102815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.101969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" Jul 23 00:20:08.102815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.102003 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 00:20:08.102815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.102160 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 00:20:08.103309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.103077 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.104666 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.103597 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz"] Jul 23 00:20:08.104813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.104794 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.105037 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.105019 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2tkd6\"" Jul 23 00:20:08.105202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.104240 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.105365 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.104317 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 00:20:08.105977 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.105958 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 00:20:08.106139 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.106114 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.106449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.106389 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.106449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.106394 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xlr65\"" Jul 23 00:20:08.107398 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.103657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.108097 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.108078 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.108448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.108429 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 00:20:08.108954 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.108698 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 00:20:08.109414 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109398 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xg85m\"" Jul 23 00:20:08.109598 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109578 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4"] Jul 23 00:20:08.109753 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109739 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw"] Jul 23 00:20:08.109866 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109850 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp"] Jul 23 00:20:08.109941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109883 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f665b65c-w5xjk"] Jul 23 00:20:08.109941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109905 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh"] Jul 23 00:20:08.109941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109923 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8kf92"] Jul 23 00:20:08.109941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109940 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr"] Jul 23 00:20:08.110129 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.109962 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jhz52"] Jul 23 00:20:08.110129 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.110058 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.110308 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.110292 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 00:20:08.111054 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.111029 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.113418 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.113395 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 00:20:08.114075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.114057 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qj2kb"] Jul 23 00:20:08.114648 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.114586 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 23 00:20:08.114902 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.114868 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 23 00:20:08.115405 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.115365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.118503 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.118483 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 23 00:20:08.118992 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.118969 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 23 00:20:08.120233 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.119787 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 00:20:08.120454 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.120422 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 00:20:08.121918 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.121901 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz"] Jul 23 00:20:08.122648 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122627 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f6fc97d76-kshb5"] Jul 23 00:20:08.122723 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122675 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66c98f88f4-c2vjt"] Jul 23 00:20:08.122723 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122701 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc"] Jul 23 00:20:08.122932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122725 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl"] Jul 23 00:20:08.122932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122751 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wvjzl"] Jul 23 00:20:08.122932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122769 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f"] Jul 23 00:20:08.122932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122635 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.122932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.122844 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hzp88\"" Jul 23 00:20:08.123677 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.123657 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb"] Jul 23 00:20:08.124702 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.124682 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qj2kb"] Jul 23 00:20:08.125000 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.124980 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 00:20:08.125135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.124991 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 00:20:08.125135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.125006 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 00:20:08.125135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.125038 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rxnz4\"" Jul 23 00:20:08.125876 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.125859 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhz52"] Jul 23 00:20:08.162540 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/578cdcb0-4740-4a05-9466-3273fd0e990b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.162692 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162667 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578cdcb0-4740-4a05-9466-3273fd0e990b-serving-cert\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.162793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-trusted-ca\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.162793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1846d59-a9b2-4eb4-8080-471b5be1f761-tmp\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.162793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162766 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.162793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c425e-9a51-48a3-ad58-96eda4dddb66-config-volume\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.162951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162858 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6b2\" (UniqueName: \"kubernetes.io/projected/b28cf90a-3973-46bb-adff-d83bfdaad30c-kube-api-access-rp6b2\") pod \"volume-data-source-validator-78cf4c6f6-jh4tw\" (UID: \"b28cf90a-3973-46bb-adff-d83bfdaad30c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" Jul 23 00:20:08.162951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.162951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162915 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-image-registry-private-configuration\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.162951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c425e-9a51-48a3-ad58-96eda4dddb66-tmp-dir\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.163144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.162987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71d213cc-26ac-4f20-98cf-e7e458cd1e2d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d69c854f6-4h82f\" (UID: \"71d213cc-26ac-4f20-98cf-e7e458cd1e2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.163144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163026 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-trusted-ca\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.163144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5d6\" (UniqueName: \"kubernetes.io/projected/92dd474f-1612-4464-ae50-4a24c828c187-kube-api-access-mh5d6\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.163144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-ca\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.163144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163117 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-hub\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.163144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d0690d-adb1-4eee-8d63-d9608764cb4e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/417d7eb2-478f-44cd-aed6-554556aa68d8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163243 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9161329a-2d0c-4b7b-bfdf-a82226410633-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrp2s\" (UniqueName: \"kubernetes.io/projected/4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9-kube-api-access-nrp2s\") pod \"network-check-source-7bc5b6bf85-p4jcl\" (UID: \"4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163299 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-image-registry-private-configuration\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7pw\" (UniqueName: \"kubernetes.io/projected/9161329a-2d0c-4b7b-bfdf-a82226410633-kube-api-access-4d7pw\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163357 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwvx\" (UniqueName: \"kubernetes.io/projected/b4ae9351-6cb1-4305-8e63-bca4ee681df1-kube-api-access-wdwvx\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.163395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-bound-sa-token\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163422 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjdcm\" (UniqueName: \"kubernetes.io/projected/a40c425e-9a51-48a3-ad58-96eda4dddb66-kube-api-access-rjdcm\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163450 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8k74\" (UniqueName: \"kubernetes.io/projected/71d213cc-26ac-4f20-98cf-e7e458cd1e2d-kube-api-access-f8k74\") pod \"managed-serviceaccount-addon-agent-d69c854f6-4h82f\" (UID: \"71d213cc-26ac-4f20-98cf-e7e458cd1e2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-stats-auth\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-installation-pull-secrets\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163611 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163790 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7mm\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-kube-api-access-hf7mm\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.163858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163842 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/578cdcb0-4740-4a05-9466-3273fd0e990b-service-ca-bundle\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92dd474f-1612-4464-ae50-4a24c828c187-config\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.163987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-bound-sa-token\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164036 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd474f-1612-4464-ae50-4a24c828c187-trusted-ca\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164064 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfk8\" (UniqueName: \"kubernetes.io/projected/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-kube-api-access-xqfk8\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-default-certificate\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75028301-5e65-4c43-8b06-28b6454f9820-ca-trust-extracted\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.164254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164230 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-installation-pull-secrets\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8vk\" (UniqueName: \"kubernetes.io/projected/578cdcb0-4740-4a05-9466-3273fd0e990b-kube-api-access-dv8vk\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164282 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164307 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2tk\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-kube-api-access-kb2tk\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164356 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxjm\" (UniqueName: \"kubernetes.io/projected/98d0690d-adb1-4eee-8d63-d9608764cb4e-kube-api-access-4mxjm\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/578cdcb0-4740-4a05-9466-3273fd0e990b-snapshots\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164406 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-registry-certificates\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164546 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164589 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfdk\" (UniqueName: \"kubernetes.io/projected/48d9735a-9a29-47c3-a62c-6626b8f341b2-kube-api-access-7nfdk\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164629 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92dd474f-1612-4464-ae50-4a24c828c187-serving-cert\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.164689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkmj\" (UniqueName: \"kubernetes.io/projected/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-kube-api-access-hbkmj\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164683 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmxk\" (UniqueName: \"kubernetes.io/projected/b1846d59-a9b2-4eb4-8080-471b5be1f761-kube-api-access-sxmxk\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164725 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164752 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-registry-certificates\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1846d59-a9b2-4eb4-8080-471b5be1f761-klusterlet-config\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.164926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/48d9735a-9a29-47c3-a62c-6626b8f341b2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.165013 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4c4590-1220-4182-b2d7-d07077143423-ca-trust-extracted\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.165043 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4ae9351-6cb1-4305-8e63-bca4ee681df1-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.165068 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4ae9351-6cb1-4305-8e63-bca4ee681df1-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.165155 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9161329a-2d0c-4b7b-bfdf-a82226410633-config\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.165189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.165188 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/578cdcb0-4740-4a05-9466-3273fd0e990b-tmp\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.266541 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrp2s\" (UniqueName: \"kubernetes.io/projected/4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9-kube-api-access-nrp2s\") pod \"network-check-source-7bc5b6bf85-p4jcl\" (UID: \"4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" Jul 23 00:20:08.266668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-image-registry-private-configuration\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.266668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7pw\" (UniqueName: \"kubernetes.io/projected/9161329a-2d0c-4b7b-bfdf-a82226410633-kube-api-access-4d7pw\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.266668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwvx\" (UniqueName: \"kubernetes.io/projected/b4ae9351-6cb1-4305-8e63-bca4ee681df1-kube-api-access-wdwvx\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.266668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-bound-sa-token\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.266872 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266702 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjdcm\" (UniqueName: \"kubernetes.io/projected/a40c425e-9a51-48a3-ad58-96eda4dddb66-kube-api-access-rjdcm\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.266984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266875 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8k74\" (UniqueName: \"kubernetes.io/projected/71d213cc-26ac-4f20-98cf-e7e458cd1e2d-kube-api-access-f8k74\") pod \"managed-serviceaccount-addon-agent-d69c854f6-4h82f\" (UID: \"71d213cc-26ac-4f20-98cf-e7e458cd1e2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.266984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-stats-auth\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.266984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-installation-pull-secrets\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.266984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266960 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.266985 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7mm\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-kube-api-access-hf7mm\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/578cdcb0-4740-4a05-9466-3273fd0e990b-service-ca-bundle\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267035 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267058 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92dd474f-1612-4464-ae50-4a24c828c187-config\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-bound-sa-token\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd474f-1612-4464-ae50-4a24c828c187-trusted-ca\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267125 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfk8\" (UniqueName: \"kubernetes.io/projected/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-kube-api-access-xqfk8\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-default-certificate\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.267182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75028301-5e65-4c43-8b06-28b6454f9820-ca-trust-extracted\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w95z\" (UniqueName: \"kubernetes.io/projected/7e966968-8137-444e-aaf0-e811d4fff488-kube-api-access-6w95z\") pod \"ingress-canary-qj2kb\" (UID: \"7e966968-8137-444e-aaf0-e811d4fff488\") " pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-installation-pull-secrets\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8vk\" (UniqueName: \"kubernetes.io/projected/578cdcb0-4740-4a05-9466-3273fd0e990b-kube-api-access-dv8vk\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267292 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2tk\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-kube-api-access-kb2tk\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.267313 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267320 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxjm\" (UniqueName: \"kubernetes.io/projected/98d0690d-adb1-4eee-8d63-d9608764cb4e-kube-api-access-4mxjm\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.267374 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls podName:6aa4ae49-ef3b-49c5-8267-fcd94a9127fa nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.767354163 +0000 UTC m=+38.064049129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-gfnsp" (UID: "6aa4ae49-ef3b-49c5-8267-fcd94a9127fa") : secret "samples-operator-tls" not found Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/578cdcb0-4740-4a05-9466-3273fd0e990b-snapshots\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-registry-certificates\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.267627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267572 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfdk\" (UniqueName: \"kubernetes.io/projected/48d9735a-9a29-47c3-a62c-6626b8f341b2-kube-api-access-7nfdk\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92dd474f-1612-4464-ae50-4a24c828c187-serving-cert\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.267623 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkmj\" (UniqueName: \"kubernetes.io/projected/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-kube-api-access-hbkmj\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.267640 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f665b65c-w5xjk: secret "image-registry-tls" not found Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.267718 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls podName:75028301-5e65-4c43-8b06-28b6454f9820 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.76770166 +0000 UTC m=+38.064396623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls") pod "image-registry-7f665b65c-w5xjk" (UID: "75028301-5e65-4c43-8b06-28b6454f9820") : secret "image-registry-tls" not found Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmxk\" (UniqueName: \"kubernetes.io/projected/b1846d59-a9b2-4eb4-8080-471b5be1f761-kube-api-access-sxmxk\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.268229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.267764 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268375 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-registry-certificates\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268422 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-registry-certificates\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1846d59-a9b2-4eb4-8080-471b5be1f761-klusterlet-config\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/48d9735a-9a29-47c3-a62c-6626b8f341b2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4c4590-1220-4182-b2d7-d07077143423-ca-trust-extracted\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4ae9351-6cb1-4305-8e63-bca4ee681df1-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4ae9351-6cb1-4305-8e63-bca4ee681df1-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.268636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9161329a-2d0c-4b7b-bfdf-a82226410633-config\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.269002 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268659 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e966968-8137-444e-aaf0-e811d4fff488-cert\") pod \"ingress-canary-qj2kb\" (UID: \"7e966968-8137-444e-aaf0-e811d4fff488\") " pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.269002 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/578cdcb0-4740-4a05-9466-3273fd0e990b-service-ca-bundle\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.269002 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.268760 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:20:08.269002 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.268799 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls podName:a40c425e-9a51-48a3-ad58-96eda4dddb66 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.768785996 +0000 UTC m=+38.065480966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls") pod "dns-default-jhz52" (UID: "a40c425e-9a51-48a3-ad58-96eda4dddb66") : secret "dns-default-metrics-tls" not found Jul 23 00:20:08.269002 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268811 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75028301-5e65-4c43-8b06-28b6454f9820-ca-trust-extracted\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.269343 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.269318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/578cdcb0-4740-4a05-9466-3273fd0e990b-snapshots\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.269449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.269429 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/578cdcb0-4740-4a05-9466-3273fd0e990b-tmp\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.269611 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.269585 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 00:20:08.272288 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.271741 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert podName:417d7eb2-478f-44cd-aed6-554556aa68d8 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.771707119 +0000 UTC m=+38.068402076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxpl4" (UID: "417d7eb2-478f-44cd-aed6-554556aa68d8") : secret "networking-console-plugin-cert" not found Jul 23 00:20:08.272288 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.272129 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:08.272288 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.272152 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c98f88f4-c2vjt: secret "image-registry-tls" not found Jul 23 00:20:08.272288 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.272216 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls podName:ae4c4590-1220-4182-b2d7-d07077143423 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.772193486 +0000 UTC m=+38.068888451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls") pod "image-registry-66c98f88f4-c2vjt" (UID: "ae4c4590-1220-4182-b2d7-d07077143423") : secret "image-registry-tls" not found Jul 23 00:20:08.272562 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.268689 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/578cdcb0-4740-4a05-9466-3273fd0e990b-tmp\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.272622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/578cdcb0-4740-4a05-9466-3273fd0e990b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.272678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578cdcb0-4740-4a05-9466-3273fd0e990b-serving-cert\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.272678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-trusted-ca\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.272785 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1846d59-a9b2-4eb4-8080-471b5be1f761-tmp\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.272785 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.272785 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c425e-9a51-48a3-ad58-96eda4dddb66-config-volume\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.272929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6b2\" (UniqueName: \"kubernetes.io/projected/b28cf90a-3973-46bb-adff-d83bfdaad30c-kube-api-access-rp6b2\") pod \"volume-data-source-validator-78cf4c6f6-jh4tw\" (UID: \"b28cf90a-3973-46bb-adff-d83bfdaad30c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" Jul 23 00:20:08.272929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.272929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-image-registry-private-configuration\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.272929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272894 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c425e-9a51-48a3-ad58-96eda4dddb66-tmp-dir\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.272929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272906 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92dd474f-1612-4464-ae50-4a24c828c187-config\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71d213cc-26ac-4f20-98cf-e7e458cd1e2d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d69c854f6-4h82f\" (UID: \"71d213cc-26ac-4f20-98cf-e7e458cd1e2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.272958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-trusted-ca\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273013 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5d6\" (UniqueName: \"kubernetes.io/projected/92dd474f-1612-4464-ae50-4a24c828c187-kube-api-access-mh5d6\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-ca\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-hub\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d0690d-adb1-4eee-8d63-d9608764cb4e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.273193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273166 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.273552 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/417d7eb2-478f-44cd-aed6-554556aa68d8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.273552 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.273231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9161329a-2d0c-4b7b-bfdf-a82226410633-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.275616 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.274395 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1846d59-a9b2-4eb4-8080-471b5be1f761-tmp\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.275616 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.275113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c425e-9a51-48a3-ad58-96eda4dddb66-tmp-dir\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.275616 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.275159 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/578cdcb0-4740-4a05-9466-3273fd0e990b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.276293 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.276255 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/48d9735a-9a29-47c3-a62c-6626b8f341b2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.277775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.277751 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-trusted-ca\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.278105 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.278084 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92dd474f-1612-4464-ae50-4a24c828c187-serving-cert\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.278615 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.278597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-stats-auth\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.279007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.278974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-default-certificate\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.279281 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.279263 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.779239976 +0000 UTC m=+38.075934939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : configmap references non-existent config key: service-ca.crt Jul 23 00:20:08.279753 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.279732 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9161329a-2d0c-4b7b-bfdf-a82226410633-config\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.279968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.279946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/417d7eb2-478f-44cd-aed6-554556aa68d8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.280050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.279946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4c4590-1220-4182-b2d7-d07077143423-ca-trust-extracted\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.280106 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.280083 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:08.280162 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.280136 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls podName:48d9735a-9a29-47c3-a62c-6626b8f341b2 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.780114336 +0000 UTC m=+38.076809300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-ndsgc" (UID: "48d9735a-9a29-47c3-a62c-6626b8f341b2") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:08.282202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.280409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578cdcb0-4740-4a05-9466-3273fd0e990b-serving-cert\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.282202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.280819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d0690d-adb1-4eee-8d63-d9608764cb4e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.282202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.281192 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4ae9351-6cb1-4305-8e63-bca4ee681df1-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.282202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.281460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-trusted-ca\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.282202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.281579 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c425e-9a51-48a3-ad58-96eda4dddb66-config-volume\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.282202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.282152 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71d213cc-26ac-4f20-98cf-e7e458cd1e2d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d69c854f6-4h82f\" (UID: \"71d213cc-26ac-4f20-98cf-e7e458cd1e2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.283343 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.283313 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd474f-1612-4464-ae50-4a24c828c187-trusted-ca\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.283553 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.283531 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.283951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.283930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-registry-certificates\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.284306 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.284284 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4ae9351-6cb1-4305-8e63-bca4ee681df1-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.284420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.284398 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-installation-pull-secrets\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.284826 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.284791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-image-registry-private-configuration\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.285560 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.285480 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:20:08.285560 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.285540 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:08.785524985 +0000 UTC m=+38.082219934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : secret "router-metrics-certs-default" not found Jul 23 00:20:08.288580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.285229 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.288813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.288788 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-ca\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.288969 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.288947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1846d59-a9b2-4eb4-8080-471b5be1f761-klusterlet-config\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.289566 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.289542 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmxk\" (UniqueName: \"kubernetes.io/projected/b1846d59-a9b2-4eb4-8080-471b5be1f761-kube-api-access-sxmxk\") pod \"klusterlet-addon-workmgr-678c68d5f7-xf5qr\" (UID: \"b1846d59-a9b2-4eb4-8080-471b5be1f761\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.290024 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.290002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-image-registry-private-configuration\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.290880 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.290409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjdcm\" (UniqueName: \"kubernetes.io/projected/a40c425e-9a51-48a3-ad58-96eda4dddb66-kube-api-access-rjdcm\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.291260 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.291223 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-bound-sa-token\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.292100 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.291551 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-bound-sa-token\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.292100 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.291900 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9161329a-2d0c-4b7b-bfdf-a82226410633-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.292100 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.292044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/98d0690d-adb1-4eee-8d63-d9608764cb4e-hub\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.292330 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.292309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwvx\" (UniqueName: \"kubernetes.io/projected/b4ae9351-6cb1-4305-8e63-bca4ee681df1-kube-api-access-wdwvx\") pod \"kube-storage-version-migrator-operator-7658d9b749-mc4fh\" (UID: \"b4ae9351-6cb1-4305-8e63-bca4ee681df1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.293293 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.292995 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7pw\" (UniqueName: \"kubernetes.io/projected/9161329a-2d0c-4b7b-bfdf-a82226410633-kube-api-access-4d7pw\") pod \"service-ca-operator-5bcbc94c78-2drrb\" (UID: \"9161329a-2d0c-4b7b-bfdf-a82226410633\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.293293 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.293000 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkmj\" (UniqueName: \"kubernetes.io/projected/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-kube-api-access-hbkmj\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.293589 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.293543 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7mm\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-kube-api-access-hf7mm\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.294400 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.294374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-installation-pull-secrets\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.296572 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.296532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5d6\" (UniqueName: \"kubernetes.io/projected/92dd474f-1612-4464-ae50-4a24c828c187-kube-api-access-mh5d6\") pod \"console-operator-565b7b569d-wvjzl\" (UID: \"92dd474f-1612-4464-ae50-4a24c828c187\") " pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.296572 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.296536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8vk\" (UniqueName: \"kubernetes.io/projected/578cdcb0-4740-4a05-9466-3273fd0e990b-kube-api-access-dv8vk\") pod \"insights-operator-555d9d7885-8kf92\" (UID: \"578cdcb0-4740-4a05-9466-3273fd0e990b\") " pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.296780 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.296673 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2tk\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-kube-api-access-kb2tk\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.296844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.296823 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxjm\" (UniqueName: \"kubernetes.io/projected/98d0690d-adb1-4eee-8d63-d9608764cb4e-kube-api-access-4mxjm\") pod \"cluster-proxy-proxy-agent-6697599ff6-v22mz\" (UID: \"98d0690d-adb1-4eee-8d63-d9608764cb4e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.297883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.297839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfdk\" (UniqueName: \"kubernetes.io/projected/48d9735a-9a29-47c3-a62c-6626b8f341b2-kube-api-access-7nfdk\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.298151 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.298130 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrp2s\" (UniqueName: \"kubernetes.io/projected/4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9-kube-api-access-nrp2s\") pod \"network-check-source-7bc5b6bf85-p4jcl\" (UID: \"4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" Jul 23 00:20:08.298457 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.298308 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfk8\" (UniqueName: \"kubernetes.io/projected/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-kube-api-access-xqfk8\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.298721 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.298681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8k74\" (UniqueName: \"kubernetes.io/projected/71d213cc-26ac-4f20-98cf-e7e458cd1e2d-kube-api-access-f8k74\") pod \"managed-serviceaccount-addon-agent-d69c854f6-4h82f\" (UID: \"71d213cc-26ac-4f20-98cf-e7e458cd1e2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.323135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.323105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6b2\" (UniqueName: \"kubernetes.io/projected/b28cf90a-3973-46bb-adff-d83bfdaad30c-kube-api-access-rp6b2\") pod \"volume-data-source-validator-78cf4c6f6-jh4tw\" (UID: \"b28cf90a-3973-46bb-adff-d83bfdaad30c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" Jul 23 00:20:08.373940 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.373916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w95z\" (UniqueName: \"kubernetes.io/projected/7e966968-8137-444e-aaf0-e811d4fff488-kube-api-access-6w95z\") pod \"ingress-canary-qj2kb\" (UID: \"7e966968-8137-444e-aaf0-e811d4fff488\") " pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.374084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.374027 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e966968-8137-444e-aaf0-e811d4fff488-cert\") pod \"ingress-canary-qj2kb\" (UID: \"7e966968-8137-444e-aaf0-e811d4fff488\") " pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.376715 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.376690 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e966968-8137-444e-aaf0-e811d4fff488-cert\") pod \"ingress-canary-qj2kb\" (UID: \"7e966968-8137-444e-aaf0-e811d4fff488\") " pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.403957 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.403933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w95z\" (UniqueName: \"kubernetes.io/projected/7e966968-8137-444e-aaf0-e811d4fff488-kube-api-access-6w95z\") pod \"ingress-canary-qj2kb\" (UID: \"7e966968-8137-444e-aaf0-e811d4fff488\") " pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.405841 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.405819 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:08.406179 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.405985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:08.418249 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.418232 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" Jul 23 00:20:08.427153 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.427137 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-8kf92" Jul 23 00:20:08.438691 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.438673 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2ns5k\"" Jul 23 00:20:08.448161 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.448140 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" Jul 23 00:20:08.454803 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.454778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" Jul 23 00:20:08.457916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.457900 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:20:08.467594 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.467567 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:08.474324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.474309 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" Jul 23 00:20:08.481313 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.481293 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:08.502340 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.502322 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" Jul 23 00:20:08.516994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.516935 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:20:08.528959 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.528907 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qj2kb" Jul 23 00:20:08.777687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.777654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:08.777896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.777720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:08.777896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.777753 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:08.777896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.777779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:08.777896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.777810 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:08.777896 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777825 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:20:08.777896 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777897 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls podName:6aa4ae49-ef3b-49c5-8267-fcd94a9127fa nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.77787732 +0000 UTC m=+39.074572285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-gfnsp" (UID: "6aa4ae49-ef3b-49c5-8267-fcd94a9127fa") : secret "samples-operator-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777902 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777905 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777932 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777965 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777969 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert podName:417d7eb2-478f-44cd-aed6-554556aa68d8 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.777952091 +0000 UTC m=+39.074647041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxpl4" (UID: "417d7eb2-478f-44cd-aed6-554556aa68d8") : secret "networking-console-plugin-cert" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777976 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f665b65c-w5xjk: secret "image-registry-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777989 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls podName:a40c425e-9a51-48a3-ad58-96eda4dddb66 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.77797892 +0000 UTC m=+39.074673869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls") pod "dns-default-jhz52" (UID: "a40c425e-9a51-48a3-ad58-96eda4dddb66") : secret "dns-default-metrics-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.778014 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls podName:75028301-5e65-4c43-8b06-28b6454f9820 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.778002268 +0000 UTC m=+39.074697219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls") pod "image-registry-7f665b65c-w5xjk" (UID: "75028301-5e65-4c43-8b06-28b6454f9820") : secret "image-registry-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.777915 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c98f88f4-c2vjt: secret "image-registry-tls" not found Jul 23 00:20:08.778151 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.778068 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls podName:ae4c4590-1220-4182-b2d7-d07077143423 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.778055988 +0000 UTC m=+39.074750937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls") pod "image-registry-66c98f88f4-c2vjt" (UID: "ae4c4590-1220-4182-b2d7-d07077143423") : secret "image-registry-tls" not found Jul 23 00:20:08.878925 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.878851 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.879065 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.879014 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:08.879065 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.879041 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.879018365 +0000 UTC m=+39.175713318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : configmap references non-existent config key: service-ca.crt Jul 23 00:20:08.879166 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.879095 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:08.879166 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.879141 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls podName:48d9735a-9a29-47c3-a62c-6626b8f341b2 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.879126848 +0000 UTC m=+39.175821812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-ndsgc" (UID: "48d9735a-9a29-47c3-a62c-6626b8f341b2") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:08.879166 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:08.879095 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:08.879166 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.879162 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:20:08.879312 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:08.879197 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:09.879185535 +0000 UTC m=+39.175880485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : secret "router-metrics-certs-default" not found Jul 23 00:20:09.405903 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.405873 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:09.408097 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.408074 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:20:09.408223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.408154 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52cgs\"" Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.792886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.793183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.793223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.793251 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793281 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793334 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793345 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls podName:6aa4ae49-ef3b-49c5-8267-fcd94a9127fa nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.793327037 +0000 UTC m=+41.090021986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-gfnsp" (UID: "6aa4ae49-ef3b-49c5-8267-fcd94a9127fa") : secret "samples-operator-tls" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.793282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793367 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls podName:a40c425e-9a51-48a3-ad58-96eda4dddb66 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.793356497 +0000 UTC m=+41.090051444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls") pod "dns-default-jhz52" (UID: "a40c425e-9a51-48a3-ad58-96eda4dddb66") : secret "dns-default-metrics-tls" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793529 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793566 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert podName:417d7eb2-478f-44cd-aed6-554556aa68d8 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.793554702 +0000 UTC m=+41.090249650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxpl4" (UID: "417d7eb2-478f-44cd-aed6-554556aa68d8") : secret "networking-console-plugin-cert" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793624 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:09.793922 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793634 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c98f88f4-c2vjt: secret "image-registry-tls" not found Jul 23 00:20:09.794707 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793662 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls podName:ae4c4590-1220-4182-b2d7-d07077143423 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.793653199 +0000 UTC m=+41.090348163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls") pod "image-registry-66c98f88f4-c2vjt" (UID: "ae4c4590-1220-4182-b2d7-d07077143423") : secret "image-registry-tls" not found Jul 23 00:20:09.794707 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793711 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:09.794707 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793718 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f665b65c-w5xjk: secret "image-registry-tls" not found Jul 23 00:20:09.794707 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.793740 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls podName:75028301-5e65-4c43-8b06-28b6454f9820 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.793732522 +0000 UTC m=+41.090427482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls") pod "image-registry-7f665b65c-w5xjk" (UID: "75028301-5e65-4c43-8b06-28b6454f9820") : secret "image-registry-tls" not found Jul 23 00:20:09.890916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.888633 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl"] Jul 23 00:20:09.894533 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.894124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:09.894533 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.894177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:09.894533 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.894227 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:09.894716 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.894562 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.894539586 +0000 UTC m=+41.191234551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : configmap references non-existent config key: service-ca.crt Jul 23 00:20:09.895184 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.894998 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:09.895184 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.895059 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls podName:48d9735a-9a29-47c3-a62c-6626b8f341b2 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.895043138 +0000 UTC m=+41.191738100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-ndsgc" (UID: "48d9735a-9a29-47c3-a62c-6626b8f341b2") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:09.895184 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.895122 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:20:09.895184 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:09.895157 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:11.895145394 +0000 UTC m=+41.191840341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : secret "router-metrics-certs-default" not found Jul 23 00:20:09.898138 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.898107 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr"] Jul 23 00:20:09.908184 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.906164 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8kf92"] Jul 23 00:20:09.909523 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.909086 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qj2kb"] Jul 23 00:20:09.920838 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.920764 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh"] Jul 23 00:20:09.924116 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.924092 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wvjzl"] Jul 23 00:20:09.927154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.927133 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz"] Jul 23 00:20:09.929932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.929912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb"] Jul 23 00:20:09.942824 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.942724 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw"] Jul 23 00:20:09.943946 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:09.943735 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f"] Jul 23 00:20:09.946308 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:09.946280 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578cdcb0_4740_4a05_9466_3273fd0e990b.slice/crio-b2b70f69ac6da896e032e8172851a294eea4d1f73d38b2daa794da7e78da482e WatchSource:0}: Error finding container b2b70f69ac6da896e032e8172851a294eea4d1f73d38b2daa794da7e78da482e: Status 404 returned error can't find the container with id b2b70f69ac6da896e032e8172851a294eea4d1f73d38b2daa794da7e78da482e Jul 23 00:20:09.948096 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:09.948046 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98d0690d_adb1_4eee_8d63_d9608764cb4e.slice/crio-20ca1fa40d7a758a9227a4e6279b3fa546672e9f5a0aef781919539b45ea16dc WatchSource:0}: Error finding container 20ca1fa40d7a758a9227a4e6279b3fa546672e9f5a0aef781919539b45ea16dc: Status 404 returned error can't find the container with id 20ca1fa40d7a758a9227a4e6279b3fa546672e9f5a0aef781919539b45ea16dc Jul 23 00:20:09.991292 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:09.991258 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ae9351_6cb1_4305_8e63_bca4ee681df1.slice/crio-611b2d53f32c3304d31df3504f8d8eca7d9d98c52ed474322779844b0002433d WatchSource:0}: Error finding container 611b2d53f32c3304d31df3504f8d8eca7d9d98c52ed474322779844b0002433d: Status 404 returned error can't find the container with id 611b2d53f32c3304d31df3504f8d8eca7d9d98c52ed474322779844b0002433d Jul 23 00:20:09.991685 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:09.991661 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28cf90a_3973_46bb_adff_d83bfdaad30c.slice/crio-9c73a7eea6e16efc15f6ef33d6e3b4e2ef1f4967c65ecb1ead55ca03bc4f94ea WatchSource:0}: Error finding container 9c73a7eea6e16efc15f6ef33d6e3b4e2ef1f4967c65ecb1ead55ca03bc4f94ea: Status 404 returned error can't find the container with id 9c73a7eea6e16efc15f6ef33d6e3b4e2ef1f4967c65ecb1ead55ca03bc4f94ea Jul 23 00:20:09.991941 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:09.991916 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9fa6f9_2cc6_43f3_a5b0_bff224dc26e9.slice/crio-c42426aa130bc4957e3f7564d6fd94a9e60b5cd6006c0d8329ea510eff05e00c WatchSource:0}: Error finding container c42426aa130bc4957e3f7564d6fd94a9e60b5cd6006c0d8329ea510eff05e00c: Status 404 returned error can't find the container with id c42426aa130bc4957e3f7564d6fd94a9e60b5cd6006c0d8329ea510eff05e00c Jul 23 00:20:09.992539 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:09.992516 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1846d59_a9b2_4eb4_8080_471b5be1f761.slice/crio-f7356ed61b85925e6b8885cd9bf3574a3d638eb04df0beac152751d920d8cb22 WatchSource:0}: Error finding container f7356ed61b85925e6b8885cd9bf3574a3d638eb04df0beac152751d920d8cb22: Status 404 returned error can't find the container with id f7356ed61b85925e6b8885cd9bf3574a3d638eb04df0beac152751d920d8cb22 Jul 23 00:20:10.253749 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.253672 2585 generic.go:356] "Generic (PLEG): container finished" podID="85950f20-fa7b-4aa1-9767-2dde5fd203ef" containerID="0596c28e383955cc5829995e05e6de142c09da64afaff849d4a8545b9206f017" exitCode=0 Jul 23 00:20:10.253903 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.253747 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerDied","Data":"0596c28e383955cc5829995e05e6de142c09da64afaff849d4a8545b9206f017"} Jul 23 00:20:10.254974 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.254954 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" event={"ID":"92dd474f-1612-4464-ae50-4a24c828c187","Type":"ContainerStarted","Data":"c0bbedf09f818c7356d3c56d430e6cdc1261741b4cc5082ad56743ff1fa2b9b2"} Jul 23 00:20:10.256067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.256039 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" event={"ID":"4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9","Type":"ContainerStarted","Data":"c42426aa130bc4957e3f7564d6fd94a9e60b5cd6006c0d8329ea510eff05e00c"} Jul 23 00:20:10.257137 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.257114 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" event={"ID":"b1846d59-a9b2-4eb4-8080-471b5be1f761","Type":"ContainerStarted","Data":"f7356ed61b85925e6b8885cd9bf3574a3d638eb04df0beac152751d920d8cb22"} Jul 23 00:20:10.258109 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.258091 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8kf92" event={"ID":"578cdcb0-4740-4a05-9466-3273fd0e990b","Type":"ContainerStarted","Data":"b2b70f69ac6da896e032e8172851a294eea4d1f73d38b2daa794da7e78da482e"} Jul 23 00:20:10.259243 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.259218 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" event={"ID":"71d213cc-26ac-4f20-98cf-e7e458cd1e2d","Type":"ContainerStarted","Data":"46af8c417e3526e1fce62068f6cda3852ca82d6683041badf432c780f6a6b197"} Jul 23 00:20:10.260372 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.260351 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qj2kb" event={"ID":"7e966968-8137-444e-aaf0-e811d4fff488","Type":"ContainerStarted","Data":"949b3ad440477cfe066c093032b329a91fcf055cb463b0384141dbf18e5b13cb"} Jul 23 00:20:10.261499 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.261459 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" event={"ID":"9161329a-2d0c-4b7b-bfdf-a82226410633","Type":"ContainerStarted","Data":"d349dd5dfedfdbb823831409ceb004443428190cee8dab8c1c3f67d76058e667"} Jul 23 00:20:10.262568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.262549 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" event={"ID":"b4ae9351-6cb1-4305-8e63-bca4ee681df1","Type":"ContainerStarted","Data":"611b2d53f32c3304d31df3504f8d8eca7d9d98c52ed474322779844b0002433d"} Jul 23 00:20:10.263564 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.263537 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" event={"ID":"b28cf90a-3973-46bb-adff-d83bfdaad30c","Type":"ContainerStarted","Data":"9c73a7eea6e16efc15f6ef33d6e3b4e2ef1f4967c65ecb1ead55ca03bc4f94ea"} Jul 23 00:20:10.264443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.264425 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" event={"ID":"98d0690d-adb1-4eee-8d63-d9608764cb4e","Type":"ContainerStarted","Data":"20ca1fa40d7a758a9227a4e6279b3fa546672e9f5a0aef781919539b45ea16dc"} Jul 23 00:20:10.281637 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.281609 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:10.384797 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.384765 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:10.444738 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.444712 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:10.546353 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.546249 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:10.601433 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:10.601401 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.229929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.229732 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.277744 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.277707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.284324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.284293 2585 generic.go:356] "Generic (PLEG): container finished" podID="85950f20-fa7b-4aa1-9767-2dde5fd203ef" containerID="9a9f4bd17e48a81978bf9cebee43a84c0ad1e3faedf9b344c544abf217111e71" exitCode=0 Jul 23 00:20:11.284440 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.284379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerDied","Data":"9a9f4bd17e48a81978bf9cebee43a84c0ad1e3faedf9b344c544abf217111e71"} Jul 23 00:20:11.492006 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.491343 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.543982 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.543945 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.617617 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.617582 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.671442 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.671376 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:11.817057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.816972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:11.817057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.817023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:11.817057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.817053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:11.817334 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.817090 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:11.817334 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.817217 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:11.817508 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.817362 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:20:11.817508 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.817429 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls podName:6aa4ae49-ef3b-49c5-8267-fcd94a9127fa nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.817408616 +0000 UTC m=+45.114103624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-gfnsp" (UID: "6aa4ae49-ef3b-49c5-8267-fcd94a9127fa") : secret "samples-operator-tls" not found Jul 23 00:20:11.817889 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.817868 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 00:20:11.817991 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.817928 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert podName:417d7eb2-478f-44cd-aed6-554556aa68d8 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.81791154 +0000 UTC m=+45.114606487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxpl4" (UID: "417d7eb2-478f-44cd-aed6-554556aa68d8") : secret "networking-console-plugin-cert" not found Jul 23 00:20:11.818065 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.817994 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:11.818065 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818005 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c98f88f4-c2vjt: secret "image-registry-tls" not found Jul 23 00:20:11.818065 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818040 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls podName:ae4c4590-1220-4182-b2d7-d07077143423 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.818027663 +0000 UTC m=+45.114722613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls") pod "image-registry-66c98f88f4-c2vjt" (UID: "ae4c4590-1220-4182-b2d7-d07077143423") : secret "image-registry-tls" not found Jul 23 00:20:11.818214 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818096 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:11.818214 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818106 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f665b65c-w5xjk: secret "image-registry-tls" not found Jul 23 00:20:11.818214 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818135 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls podName:75028301-5e65-4c43-8b06-28b6454f9820 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.818124475 +0000 UTC m=+45.114819421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls") pod "image-registry-7f665b65c-w5xjk" (UID: "75028301-5e65-4c43-8b06-28b6454f9820") : secret "image-registry-tls" not found Jul 23 00:20:11.818214 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818189 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:20:11.818400 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.818220 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls podName:a40c425e-9a51-48a3-ad58-96eda4dddb66 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.818208406 +0000 UTC m=+45.114903354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls") pod "dns-default-jhz52" (UID: "a40c425e-9a51-48a3-ad58-96eda4dddb66") : secret "dns-default-metrics-tls" not found Jul 23 00:20:11.918082 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.918045 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:11.918244 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.918099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:11.918244 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:11.918146 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:11.918363 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.918295 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.918275997 +0000 UTC m=+45.214970946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : configmap references non-existent config key: service-ca.crt Jul 23 00:20:11.918740 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.918719 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:11.918814 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.918782 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls podName:48d9735a-9a29-47c3-a62c-6626b8f341b2 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.918764943 +0000 UTC m=+45.215459896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-ndsgc" (UID: "48d9735a-9a29-47c3-a62c-6626b8f341b2") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:11.918884 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.918840 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:20:11.918884 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:11.918872 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:15.918861964 +0000 UTC m=+45.215556915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : secret "router-metrics-certs-default" not found Jul 23 00:20:12.229529 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.229402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:12.302304 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.301439 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" event={"ID":"85950f20-fa7b-4aa1-9767-2dde5fd203ef","Type":"ContainerStarted","Data":"3a7ac589aff6ac5ddd75448a24ed120abb4e571e609a0e69a94d54b407f39a57"} Jul 23 00:20:12.325397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.325347 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:12.326978 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.325576 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vsxtl" podStartSLOduration=9.352981316 podStartE2EDuration="41.325436552s" totalImagesPullingTime="31.972455236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:12.32403666 +0000 UTC m=+41.620731630" watchObservedRunningTime="2026-07-23 00:20:12.325436552 +0000 UTC m=+41.622131532" Jul 23 00:20:12.415488 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.415403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:12.520877 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.520796 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:12.816030 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.815937 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:12.878358 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:12.878309 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:13.664445 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:13.664375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:13.766702 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:13.766107 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:14.015145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:14.015040 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:14.074647 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:14.074548 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:14.258363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:14.258302 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:14.367342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:14.367256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:15.864837 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.864793 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.864878 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.864919 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.864949 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.864980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.864995 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865059 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865073 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c98f88f4-c2vjt: secret "image-registry-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865077 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls podName:6aa4ae49-ef3b-49c5-8267-fcd94a9127fa nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.865055967 +0000 UTC m=+53.161750936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-gfnsp" (UID: "6aa4ae49-ef3b-49c5-8267-fcd94a9127fa") : secret "samples-operator-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865118 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls podName:ae4c4590-1220-4182-b2d7-d07077143423 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.865107748 +0000 UTC m=+53.161802694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls") pod "image-registry-66c98f88f4-c2vjt" (UID: "ae4c4590-1220-4182-b2d7-d07077143423") : secret "image-registry-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.864995 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865129 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865136 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865149 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert podName:417d7eb2-478f-44cd-aed6-554556aa68d8 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.865142059 +0000 UTC m=+53.161837005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxpl4" (UID: "417d7eb2-478f-44cd-aed6-554556aa68d8") : secret "networking-console-plugin-cert" not found Jul 23 00:20:15.865280 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865156 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f665b65c-w5xjk: secret "image-registry-tls" not found Jul 23 00:20:15.865831 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865179 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls podName:a40c425e-9a51-48a3-ad58-96eda4dddb66 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.865163477 +0000 UTC m=+53.161858430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls") pod "dns-default-jhz52" (UID: "a40c425e-9a51-48a3-ad58-96eda4dddb66") : secret "dns-default-metrics-tls" not found Jul 23 00:20:15.865831 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.865204 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls podName:75028301-5e65-4c43-8b06-28b6454f9820 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.865186922 +0000 UTC m=+53.161881867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls") pod "image-registry-7f665b65c-w5xjk" (UID: "75028301-5e65-4c43-8b06-28b6454f9820") : secret "image-registry-tls" not found Jul 23 00:20:15.965596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.965555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:15.965596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.965600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:15.965804 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:15.965628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:15.965804 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.965731 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:20:15.965804 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.965739 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.96572581 +0000 UTC m=+53.262420755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : configmap references non-existent config key: service-ca.crt Jul 23 00:20:15.965965 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.965813 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.965797667 +0000 UTC m=+53.262492628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : secret "router-metrics-certs-default" not found Jul 23 00:20:15.965965 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.965818 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:15.965965 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:15.965879 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls podName:48d9735a-9a29-47c3-a62c-6626b8f341b2 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:23.965866052 +0000 UTC m=+53.262560998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-ndsgc" (UID: "48d9735a-9a29-47c3-a62c-6626b8f341b2") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:23.944923 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:23.944869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:23.944923 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:23.944921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:23.944952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:23.944983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945104 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945142 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945156 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c98f88f4-c2vjt: secret "image-registry-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945159 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945168 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert podName:417d7eb2-478f-44cd-aed6-554556aa68d8 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:39.945146182 +0000 UTC m=+69.241841161 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxpl4" (UID: "417d7eb2-478f-44cd-aed6-554556aa68d8") : secret "networking-console-plugin-cert" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945191 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls podName:ae4c4590-1220-4182-b2d7-d07077143423 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:39.945178027 +0000 UTC m=+69.241872989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls") pod "image-registry-66c98f88f4-c2vjt" (UID: "ae4c4590-1220-4182-b2d7-d07077143423") : secret "image-registry-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945213 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls podName:a40c425e-9a51-48a3-ad58-96eda4dddb66 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:39.945198236 +0000 UTC m=+69.241893185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls") pod "dns-default-jhz52" (UID: "a40c425e-9a51-48a3-ad58-96eda4dddb66") : secret "dns-default-metrics-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945244 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945254 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f665b65c-w5xjk: secret "image-registry-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945284 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls podName:75028301-5e65-4c43-8b06-28b6454f9820 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:39.945272851 +0000 UTC m=+69.241967804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls") pod "image-registry-7f665b65c-w5xjk" (UID: "75028301-5e65-4c43-8b06-28b6454f9820") : secret "image-registry-tls" not found Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:23.945304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:23.945493 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945375 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:20:23.946191 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:23.945396 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls podName:6aa4ae49-ef3b-49c5-8267-fcd94a9127fa nodeName:}" failed. No retries permitted until 2026-07-23 00:20:39.945389971 +0000 UTC m=+69.242084920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-gfnsp" (UID: "6aa4ae49-ef3b-49c5-8267-fcd94a9127fa") : secret "samples-operator-tls" not found Jul 23 00:20:24.046377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.046290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:24.046377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.046325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:24.046377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.046364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:24.046662 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:24.046448 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:24.046662 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:24.046531 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls podName:48d9735a-9a29-47c3-a62c-6626b8f341b2 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:40.046512894 +0000 UTC m=+69.343207853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-ndsgc" (UID: "48d9735a-9a29-47c3-a62c-6626b8f341b2") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:20:24.046662 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:24.046549 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:40.046540014 +0000 UTC m=+69.343234960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : configmap references non-existent config key: service-ca.crt Jul 23 00:20:24.046662 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:24.046565 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:20:24.046662 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:24.046593 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs podName:5fca11ac-2e0f-4249-9bfd-060eeae9dbd3 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:40.046584305 +0000 UTC m=+69.343279254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs") pod "router-default-f6fc97d76-kshb5" (UID: "5fca11ac-2e0f-4249-9bfd-060eeae9dbd3") : secret "router-metrics-certs-default" not found Jul 23 00:20:24.341860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.341821 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8kf92" event={"ID":"578cdcb0-4740-4a05-9466-3273fd0e990b","Type":"ContainerStarted","Data":"5d9c3a61fff6038f5658f0a346d5941943a146e5066d699db947db1259883795"} Jul 23 00:20:24.344099 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.344070 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" event={"ID":"71d213cc-26ac-4f20-98cf-e7e458cd1e2d","Type":"ContainerStarted","Data":"2eeb6103a6de3689ba1a564b2cefcf1dedfac4b469666fe1b0e068d7b9682c4f"} Jul 23 00:20:24.346023 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.345990 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qj2kb" event={"ID":"7e966968-8137-444e-aaf0-e811d4fff488","Type":"ContainerStarted","Data":"cc4ffcdc1d42235770a521711450531bca6b2a535d9040d25fcdf6202a0ffa29"} Jul 23 00:20:24.347585 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.347560 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" event={"ID":"9161329a-2d0c-4b7b-bfdf-a82226410633","Type":"ContainerStarted","Data":"5b982d830bd47bd0928c37b596d7ea50353308dab17e26f05f4327699b70a164"} Jul 23 00:20:24.349531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.349502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" event={"ID":"b4ae9351-6cb1-4305-8e63-bca4ee681df1","Type":"ContainerStarted","Data":"60700de09ec9a00fc86c2ae6141364c686ba2db8056cddbd7557156d9a9f7de8"} Jul 23 00:20:24.351922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.350844 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" event={"ID":"b28cf90a-3973-46bb-adff-d83bfdaad30c","Type":"ContainerStarted","Data":"6072d39050cbe8632603daa5a4d339d72f9bd9d588a1732aa0a0f3d287914f9b"} Jul 23 00:20:24.353237 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.352876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" event={"ID":"98d0690d-adb1-4eee-8d63-d9608764cb4e","Type":"ContainerStarted","Data":"6c4ed24b415239c296fe26dd07012bdf521d29d4c1439645bc86fded68ee18fb"} Jul 23 00:20:24.353237 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.352910 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" event={"ID":"98d0690d-adb1-4eee-8d63-d9608764cb4e","Type":"ContainerStarted","Data":"fe58eb3e696c3073373f99e92d281f14e2f5ce58e52799895333f2df068357ef"} Jul 23 00:20:24.353237 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.352927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" event={"ID":"98d0690d-adb1-4eee-8d63-d9608764cb4e","Type":"ContainerStarted","Data":"3afc524bba259044b0b02c9c724073c7d6e9b45a2e7cfb2ff359bd9aaf2ff199"} Jul 23 00:20:24.354460 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.354443 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/0.log" Jul 23 00:20:24.354561 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.354493 2585 generic.go:356] "Generic (PLEG): container finished" podID="92dd474f-1612-4464-ae50-4a24c828c187" containerID="555052e89f3061fa48d34f7fe44e484c0e22b599c898bb1ef52b731ec572e7db" exitCode=255 Jul 23 00:20:24.354623 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.354567 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" event={"ID":"92dd474f-1612-4464-ae50-4a24c828c187","Type":"ContainerDied","Data":"555052e89f3061fa48d34f7fe44e484c0e22b599c898bb1ef52b731ec572e7db"} Jul 23 00:20:24.354760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.354736 2585 scope.go:117] "RemoveContainer" containerID="555052e89f3061fa48d34f7fe44e484c0e22b599c898bb1ef52b731ec572e7db" Jul 23 00:20:24.356271 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.356254 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" event={"ID":"4e9fa6f9-2cc6-43f3-a5b0-bff224dc26e9","Type":"ContainerStarted","Data":"3e9532f33fe6c7513db84478d60feb4fd0ecc5724cbeb916aca62a0876c8ddd4"} Jul 23 00:20:24.358170 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.358143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" event={"ID":"b1846d59-a9b2-4eb4-8080-471b5be1f761","Type":"ContainerStarted","Data":"a3cc470f90bdeedcfa291bfe8a1dfd5b5a50e457e76af77c5edc0e1ab7c2ee49"} Jul 23 00:20:24.358837 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.358791 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-8kf92" podStartSLOduration=15.771587409 podStartE2EDuration="29.358775093s" totalImagesPullingTime="13.587187684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.357710395 +0000 UTC m=+53.654405365" watchObservedRunningTime="2026-07-23 00:20:24.358775093 +0000 UTC m=+53.655470062" Jul 23 00:20:24.359195 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.358818 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:24.360695 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.360678 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" Jul 23 00:20:24.393355 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.393311 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" podStartSLOduration=15.816571008 podStartE2EDuration="29.393290085s" totalImagesPullingTime="13.576719077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.373083122 +0000 UTC m=+53.669778091" watchObservedRunningTime="2026-07-23 00:20:24.393290085 +0000 UTC m=+53.689985034" Jul 23 00:20:24.406755 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.406705 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qj2kb" podStartSLOduration=2.823249867 podStartE2EDuration="16.4066855s" totalImagesPullingTime="13.583435633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.405967963 +0000 UTC m=+53.702662948" watchObservedRunningTime="2026-07-23 00:20:24.4066855 +0000 UTC m=+53.703380468" Jul 23 00:20:24.424375 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.424329 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" podStartSLOduration=15.882384197 podStartE2EDuration="29.424310442s" totalImagesPullingTime="13.541926245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.423366486 +0000 UTC m=+53.720061450" watchObservedRunningTime="2026-07-23 00:20:24.424310442 +0000 UTC m=+53.721005411" Jul 23 00:20:24.447710 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.445916 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" podStartSLOduration=3.846779841 podStartE2EDuration="17.445894566s" totalImagesPullingTime="13.599114725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.444536202 +0000 UTC m=+53.741231172" watchObservedRunningTime="2026-07-23 00:20:24.445894566 +0000 UTC m=+53.742589535" Jul 23 00:20:24.462397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.462350 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-p4jcl" podStartSLOduration=15.921199421 podStartE2EDuration="29.46233146s" totalImagesPullingTime="13.541132039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.459189899 +0000 UTC m=+53.755884893" watchObservedRunningTime="2026-07-23 00:20:24.46233146 +0000 UTC m=+53.759026430" Jul 23 00:20:24.475437 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.475396 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d69c854f6-4h82f" podStartSLOduration=3.8999455579999998 podStartE2EDuration="17.475378719s" totalImagesPullingTime="13.575433161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.474167011 +0000 UTC m=+53.770861980" watchObservedRunningTime="2026-07-23 00:20:24.475378719 +0000 UTC m=+53.772073691" Jul 23 00:20:24.488775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.488237 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jh4tw" podStartSLOduration=16.561344917 podStartE2EDuration="29.488219042s" totalImagesPullingTime="12.926874125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.486619984 +0000 UTC m=+53.783314954" watchObservedRunningTime="2026-07-23 00:20:24.488219042 +0000 UTC m=+53.784914008" Jul 23 00:20:24.502290 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.502248 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-678c68d5f7-xf5qr" podStartSLOduration=3.960971626 podStartE2EDuration="17.50222975s" totalImagesPullingTime="13.541258124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:24.501290127 +0000 UTC m=+53.797985098" watchObservedRunningTime="2026-07-23 00:20:24.50222975 +0000 UTC m=+53.798924718" Jul 23 00:20:24.571055 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.570951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:24.673374 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:24.673329 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:25.143656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.143623 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:25.244576 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.244545 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:25.367034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.367007 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:20:25.367486 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.367452 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/0.log" Jul 23 00:20:25.367607 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.367510 2585 generic.go:356] "Generic (PLEG): container finished" podID="92dd474f-1612-4464-ae50-4a24c828c187" containerID="d8cd98aaae0ea7b68286b4adb89bb1be61b3df345c822182bc9e85ca2d88f6d3" exitCode=255 Jul 23 00:20:25.367667 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.367638 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" event={"ID":"92dd474f-1612-4464-ae50-4a24c828c187","Type":"ContainerDied","Data":"d8cd98aaae0ea7b68286b4adb89bb1be61b3df345c822182bc9e85ca2d88f6d3"} Jul 23 00:20:25.367746 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.367730 2585 scope.go:117] "RemoveContainer" containerID="555052e89f3061fa48d34f7fe44e484c0e22b599c898bb1ef52b731ec572e7db" Jul 23 00:20:25.367884 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.367859 2585 scope.go:117] "RemoveContainer" containerID="d8cd98aaae0ea7b68286b4adb89bb1be61b3df345c822182bc9e85ca2d88f6d3" Jul 23 00:20:25.368084 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:25.368060 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wvjzl_openshift-console-operator(92dd474f-1612-4464-ae50-4a24c828c187)\"" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" podUID="92dd474f-1612-4464-ae50-4a24c828c187" Jul 23 00:20:25.810486 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.810435 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vjbl8"] Jul 23 00:20:25.834225 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.834191 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:25.844738 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.844712 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vjbl8"] Jul 23 00:20:25.844909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.844856 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:25.846896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.846865 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 00:20:25.847030 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.846898 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t68f8\"" Jul 23 00:20:25.847030 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.847022 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 00:20:25.892986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.892938 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:25.965620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.965585 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a1deeea-5ee1-46da-8634-c754a6a24cc0-crio-socket\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:25.965820 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.965646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tkc\" (UniqueName: \"kubernetes.io/projected/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-api-access-g9tkc\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:25.965820 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.965744 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a1deeea-5ee1-46da-8634-c754a6a24cc0-data-volume\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:25.965946 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.965819 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:25.965946 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:25.965852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.066802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.066730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a1deeea-5ee1-46da-8634-c754a6a24cc0-crio-socket\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.066802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.066779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tkc\" (UniqueName: \"kubernetes.io/projected/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-api-access-g9tkc\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.066960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.066832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a1deeea-5ee1-46da-8634-c754a6a24cc0-data-volume\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.066960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.066862 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a1deeea-5ee1-46da-8634-c754a6a24cc0-crio-socket\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.067022 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.066985 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.067067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.067020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.067103 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.067091 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:20:26.067146 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.067139 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:26.56712438 +0000 UTC m=+55.863819326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : configmap "kube-rbac-proxy" not found Jul 23 00:20:26.067213 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.067196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a1deeea-5ee1-46da-8634-c754a6a24cc0-data-volume\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.067256 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.067209 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:20:26.067336 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.067268 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:26.567256007 +0000 UTC m=+55.863950957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : secret "insights-runtime-extractor-tls" not found Jul 23 00:20:26.076439 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.076409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tkc\" (UniqueName: \"kubernetes.io/projected/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-api-access-g9tkc\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.131772 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.131738 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:26.132084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.132066 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4"] Jul 23 00:20:26.153835 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.153809 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4"] Jul 23 00:20:26.154174 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.153899 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" Jul 23 00:20:26.156114 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.156083 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 00:20:26.156114 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.156097 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ds4fs\"" Jul 23 00:20:26.156273 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.156136 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 00:20:26.188577 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.188551 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:26.240136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.240112 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:26.269565 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.269544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfgq\" (UniqueName: \"kubernetes.io/projected/93a735b7-a90b-4e28-bea9-1c484551f18e-kube-api-access-jkfgq\") pod \"migrator-7cc9c7ffd7-kc5d4\" (UID: \"93a735b7-a90b-4e28-bea9-1c484551f18e\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" Jul 23 00:20:26.370712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.370649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfgq\" (UniqueName: \"kubernetes.io/projected/93a735b7-a90b-4e28-bea9-1c484551f18e-kube-api-access-jkfgq\") pod \"migrator-7cc9c7ffd7-kc5d4\" (UID: \"93a735b7-a90b-4e28-bea9-1c484551f18e\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" Jul 23 00:20:26.372861 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.372845 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:20:26.373168 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.373151 2585 scope.go:117] "RemoveContainer" containerID="d8cd98aaae0ea7b68286b4adb89bb1be61b3df345c822182bc9e85ca2d88f6d3" Jul 23 00:20:26.373341 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.373322 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wvjzl_openshift-console-operator(92dd474f-1612-4464-ae50-4a24c828c187)\"" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" podUID="92dd474f-1612-4464-ae50-4a24c828c187" Jul 23 00:20:26.378453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.378433 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfgq\" (UniqueName: \"kubernetes.io/projected/93a735b7-a90b-4e28-bea9-1c484551f18e-kube-api-access-jkfgq\") pod \"migrator-7cc9c7ffd7-kc5d4\" (UID: \"93a735b7-a90b-4e28-bea9-1c484551f18e\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" Jul 23 00:20:26.463510 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.463485 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" Jul 23 00:20:26.572339 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.572307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.572454 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.572361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:26.572454 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.572437 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:20:26.572552 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.572527 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:27.57250427 +0000 UTC m=+56.869199219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : configmap "kube-rbac-proxy" not found Jul 23 00:20:26.572592 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.572552 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:20:26.572622 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:26.572616 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:27.572599436 +0000 UTC m=+56.869294385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : secret "insights-runtime-extractor-tls" not found Jul 23 00:20:26.594026 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.593978 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:26.609058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.609031 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4"] Jul 23 00:20:26.611786 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:26.611757 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a735b7_a90b_4e28_bea9_1c484551f18e.slice/crio-67b4469c298175abe2d95d7d35e0c926a3df35b60382972c59e5b425a349bbcd WatchSource:0}: Error finding container 67b4469c298175abe2d95d7d35e0c926a3df35b60382972c59e5b425a349bbcd: Status 404 returned error can't find the container with id 67b4469c298175abe2d95d7d35e0c926a3df35b60382972c59e5b425a349bbcd Jul 23 00:20:26.694919 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:26.694836 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:27.001817 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:27.001793 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l4m8c_e1146a88-38cb-480f-af98-ead7a0b8a6b1/dns-node-resolver/0.log" Jul 23 00:20:27.378779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:27.378692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" event={"ID":"93a735b7-a90b-4e28-bea9-1c484551f18e","Type":"ContainerStarted","Data":"67b4469c298175abe2d95d7d35e0c926a3df35b60382972c59e5b425a349bbcd"} Jul 23 00:20:27.583005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:27.582969 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:27.583005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:27.583010 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:27.583202 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:27.583097 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:20:27.583202 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:27.583115 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:20:27.583202 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:27.583175 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:29.58315785 +0000 UTC m=+58.879852798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : secret "insights-runtime-extractor-tls" not found Jul 23 00:20:27.583312 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:27.583231 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:29.583216988 +0000 UTC m=+58.879911934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : configmap "kube-rbac-proxy" not found Jul 23 00:20:28.001537 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.001487 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j5x4l_aa64cb4e-3578-4705-addb-932930f73df9/node-ca/0.log" Jul 23 00:20:28.184988 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.184960 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8bwx2"] Jul 23 00:20:28.187202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.187182 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.189098 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.189073 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 00:20:28.189098 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.189085 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 00:20:28.189663 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.189647 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 00:20:28.189663 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.189648 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x2dj9\"" Jul 23 00:20:28.189781 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.189667 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 00:20:28.196341 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.196320 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8bwx2"] Jul 23 00:20:28.220008 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.219976 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:28.289879 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.289818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bb99586-615d-4fa7-82e3-08976bd7378b-signing-key\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.289879 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.289847 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bb99586-615d-4fa7-82e3-08976bd7378b-signing-cabundle\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.289999 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.289921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9mm\" (UniqueName: \"kubernetes.io/projected/4bb99586-615d-4fa7-82e3-08976bd7378b-kube-api-access-nf9mm\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.328143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.328122 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:28.383338 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.383310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" event={"ID":"93a735b7-a90b-4e28-bea9-1c484551f18e","Type":"ContainerStarted","Data":"2b90a57cc1816b015378914e78db81eab6a4ad4e1ca01c5e1f89a08e8fa76a3a"} Jul 23 00:20:28.383641 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.383359 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" event={"ID":"93a735b7-a90b-4e28-bea9-1c484551f18e","Type":"ContainerStarted","Data":"ce6d97ee732a6ea947a394c87423042ce83e272033bb211c1b27615cb6e45723"} Jul 23 00:20:28.391072 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.391051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bb99586-615d-4fa7-82e3-08976bd7378b-signing-key\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.391128 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.391080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bb99586-615d-4fa7-82e3-08976bd7378b-signing-cabundle\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.391163 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.391142 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9mm\" (UniqueName: \"kubernetes.io/projected/4bb99586-615d-4fa7-82e3-08976bd7378b-kube-api-access-nf9mm\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.391806 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.391787 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bb99586-615d-4fa7-82e3-08976bd7378b-signing-cabundle\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.393501 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.393482 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bb99586-615d-4fa7-82e3-08976bd7378b-signing-key\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.397002 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.396962 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kc5d4" podStartSLOduration=1.023172196 podStartE2EDuration="2.396946938s" totalImagesPullingTime="1.373774742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:28.396778611 +0000 UTC m=+57.693473600" watchObservedRunningTime="2026-07-23 00:20:28.396946938 +0000 UTC m=+57.693641908" Jul 23 00:20:28.401439 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.401420 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9mm\" (UniqueName: \"kubernetes.io/projected/4bb99586-615d-4fa7-82e3-08976bd7378b-kube-api-access-nf9mm\") pod \"service-ca-58598695f7-8bwx2\" (UID: \"4bb99586-615d-4fa7-82e3-08976bd7378b\") " pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.467535 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.467507 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:28.467713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.467609 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:28.467713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.467657 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:28.468059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.468044 2585 scope.go:117] "RemoveContainer" containerID="d8cd98aaae0ea7b68286b4adb89bb1be61b3df345c822182bc9e85ca2d88f6d3" Jul 23 00:20:28.468283 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:28.468256 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wvjzl_openshift-console-operator(92dd474f-1612-4464-ae50-4a24c828c187)\"" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" podUID="92dd474f-1612-4464-ae50-4a24c828c187" Jul 23 00:20:28.496965 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.496942 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-8bwx2" Jul 23 00:20:28.569351 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.569320 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:28.601905 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.601873 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qj2kb_7e966968-8137-444e-aaf0-e811d4fff488/serve-healthcheck-canary/0.log" Jul 23 00:20:28.617184 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:28.617058 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8bwx2"] Jul 23 00:20:28.619450 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:28.619430 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb99586_615d_4fa7_82e3_08976bd7378b.slice/crio-39cd324ec4ae2803e72497cfe3d0cb202e326425a5906c55bb3efaa2495089f6 WatchSource:0}: Error finding container 39cd324ec4ae2803e72497cfe3d0cb202e326425a5906c55bb3efaa2495089f6: Status 404 returned error can't find the container with id 39cd324ec4ae2803e72497cfe3d0cb202e326425a5906c55bb3efaa2495089f6 Jul 23 00:20:29.263775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.263746 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:29.390555 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.390524 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-8bwx2" event={"ID":"4bb99586-615d-4fa7-82e3-08976bd7378b","Type":"ContainerStarted","Data":"b851b4f6a150152db692130ea4c705cab2ee79080c2331a175a0a7fc4ff746ff"} Jul 23 00:20:29.391173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.391151 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-8bwx2" event={"ID":"4bb99586-615d-4fa7-82e3-08976bd7378b","Type":"ContainerStarted","Data":"39cd324ec4ae2803e72497cfe3d0cb202e326425a5906c55bb3efaa2495089f6"} Jul 23 00:20:29.417521 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.417442 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-8bwx2" podStartSLOduration=1.417423434 podStartE2EDuration="1.417423434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:29.41504903 +0000 UTC m=+58.711743999" watchObservedRunningTime="2026-07-23 00:20:29.417423434 +0000 UTC m=+58.714118402" Jul 23 00:20:29.421267 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.421198 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:29.520686 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.520604 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:29.603531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.603484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:29.603693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.603540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:29.603693 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:29.603596 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:20:29.603693 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:29.603666 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:33.603645515 +0000 UTC m=+62.900340478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : configmap "kube-rbac-proxy" not found Jul 23 00:20:29.603859 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:29.603707 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:20:29.603859 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:29.603754 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:33.603738978 +0000 UTC m=+62.900433930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : secret "insights-runtime-extractor-tls" not found Jul 23 00:20:29.668019 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.667942 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:29.769294 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.769256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:29.921532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:29.921393 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:30.108840 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:30.108800 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:31.550139 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:31.550104 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:31.649088 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:31.649049 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:31.749671 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:31.749640 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:31.911451 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:31.911357 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:32.108716 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:32.108687 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:32.377526 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:32.377491 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.104612 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.104571 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.205069 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.205036 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.249851 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.249825 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xlf8" Jul 23 00:20:33.305849 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.305815 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.505335 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.505307 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.641764 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.641731 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:33.641918 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.641777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:33.641918 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:33.641855 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:20:33.642040 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:33.641946 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:20:33.642040 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:33.641954 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:41.641933582 +0000 UTC m=+70.938628534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : configmap "kube-rbac-proxy" not found Jul 23 00:20:33.642040 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:20:33.642007 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls podName:3a1deeea-5ee1-46da-8634-c754a6a24cc0 nodeName:}" failed. No retries permitted until 2026-07-23 00:20:41.641989556 +0000 UTC m=+70.938684519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vjbl8" (UID: "3a1deeea-5ee1-46da-8634-c754a6a24cc0") : secret "insights-runtime-extractor-tls" not found Jul 23 00:20:33.710514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.710481 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.925037 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:33.924946 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:34.025580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:34.025547 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:34.127587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:34.127548 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.320566 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.320534 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.393547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.393516 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.449242 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.449215 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.507785 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.507757 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.712741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.712642 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.905428 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.905388 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:35.964423 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:35.964347 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:36.112167 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:36.112134 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:36.910975 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:36.910944 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:36.980554 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:36.980514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:36.983228 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:36.983205 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:20:36.993313 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:36.993283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac42cc77-e8f3-4ee1-a85b-03b0750efc61-metrics-certs\") pod \"network-metrics-daemon-4w22f\" (UID: \"ac42cc77-e8f3-4ee1-a85b-03b0750efc61\") " pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:37.017249 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.017226 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52cgs\"" Jul 23 00:20:37.025908 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.025888 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4w22f" Jul 23 00:20:37.081874 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.081833 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:37.084411 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.084218 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:20:37.094841 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.094820 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd3be28b-aa8d-4a5a-b230-dbedffd5543c-original-pull-secret\") pod \"global-pull-secret-syncer-spxkd\" (UID: \"cd3be28b-aa8d-4a5a-b230-dbedffd5543c\") " pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:37.148828 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.148768 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4w22f"] Jul 23 00:20:37.151118 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:37.151093 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac42cc77_e8f3_4ee1_a85b_03b0750efc61.slice/crio-33cdb9d8f96a6fe86c70f5a0062a7382adeb9e14cb9279e64c3ffdae67022fcc WatchSource:0}: Error finding container 33cdb9d8f96a6fe86c70f5a0062a7382adeb9e14cb9279e64c3ffdae67022fcc: Status 404 returned error can't find the container with id 33cdb9d8f96a6fe86c70f5a0062a7382adeb9e14cb9279e64c3ffdae67022fcc Jul 23 00:20:37.183013 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.182952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:37.185269 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.185252 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqprx\" (UniqueName: \"kubernetes.io/projected/e04e9d9a-ede9-4fa6-8857-e78e80726373-kube-api-access-nqprx\") pod \"network-check-target-gdqcl\" (UID: \"e04e9d9a-ede9-4fa6-8857-e78e80726373\") " pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:37.219394 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.219374 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2ns5k\"" Jul 23 00:20:37.222905 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.222891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxkd" Jul 23 00:20:37.228356 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.228339 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:37.353526 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.353501 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-spxkd"] Jul 23 00:20:37.356354 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:37.356324 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3be28b_aa8d_4a5a_b230_dbedffd5543c.slice/crio-5c41b9d7a3a712e35e45ecccd0ae69f273c3cca2b9ca27502aeccdc668aba970 WatchSource:0}: Error finding container 5c41b9d7a3a712e35e45ecccd0ae69f273c3cca2b9ca27502aeccdc668aba970: Status 404 returned error can't find the container with id 5c41b9d7a3a712e35e45ecccd0ae69f273c3cca2b9ca27502aeccdc668aba970 Jul 23 00:20:37.372046 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.372018 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gdqcl"] Jul 23 00:20:37.376113 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:37.376086 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e9d9a_ede9_4fa6_8857_e78e80726373.slice/crio-51d35243b10c6aee3a11387ae6f7ba9d5ba849e3f2da14343e0e46b21968ca6f WatchSource:0}: Error finding container 51d35243b10c6aee3a11387ae6f7ba9d5ba849e3f2da14343e0e46b21968ca6f: Status 404 returned error can't find the container with id 51d35243b10c6aee3a11387ae6f7ba9d5ba849e3f2da14343e0e46b21968ca6f Jul 23 00:20:37.417263 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.417233 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gdqcl" event={"ID":"e04e9d9a-ede9-4fa6-8857-e78e80726373","Type":"ContainerStarted","Data":"51d35243b10c6aee3a11387ae6f7ba9d5ba849e3f2da14343e0e46b21968ca6f"} Jul 23 00:20:37.418698 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.418433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-spxkd" event={"ID":"cd3be28b-aa8d-4a5a-b230-dbedffd5543c","Type":"ContainerStarted","Data":"5c41b9d7a3a712e35e45ecccd0ae69f273c3cca2b9ca27502aeccdc668aba970"} Jul 23 00:20:37.419615 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.419593 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4w22f" event={"ID":"ac42cc77-e8f3-4ee1-a85b-03b0750efc61","Type":"ContainerStarted","Data":"33cdb9d8f96a6fe86c70f5a0062a7382adeb9e14cb9279e64c3ffdae67022fcc"} Jul 23 00:20:37.505503 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.505453 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:37.705621 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.705537 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:37.907730 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:37.907700 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:38.109403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:38.109370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:38.425850 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:38.425736 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gdqcl" event={"ID":"e04e9d9a-ede9-4fa6-8857-e78e80726373","Type":"ContainerStarted","Data":"2cac5f35bd1390798f9ab2649b02c0df0195c4e4b4735ac784fc0d9c7f49f128"} Jul 23 00:20:38.425996 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:38.425943 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:20:38.427793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:38.427762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4w22f" event={"ID":"ac42cc77-e8f3-4ee1-a85b-03b0750efc61","Type":"ContainerStarted","Data":"c7f68c253e7250584e5738630621ee6a54f69692911141f11b92321f3f949d8c"} Jul 23 00:20:38.441019 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:38.440980 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gdqcl" podStartSLOduration=67.440967331 podStartE2EDuration="1m7.440967331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:38.439429604 +0000 UTC m=+67.736124594" watchObservedRunningTime="2026-07-23 00:20:38.440967331 +0000 UTC m=+67.737662299" Jul 23 00:20:38.911509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:38.911454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:39.089124 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:39.089076 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:39.189477 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:39.189380 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:39.433567 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:39.433493 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4w22f" event={"ID":"ac42cc77-e8f3-4ee1-a85b-03b0750efc61","Type":"ContainerStarted","Data":"667abf19b7e22a30f7eca065d87659ea22182da64ab06eecc1a2f9fe8c072655"} Jul 23 00:20:39.449983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:39.449892 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4w22f" podStartSLOduration=67.293596371 podStartE2EDuration="1m8.44987499s" totalImagesPullingTime="1.156278619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:39.447500733 +0000 UTC m=+68.744195730" watchObservedRunningTime="2026-07-23 00:20:39.44987499 +0000 UTC m=+68.746569959" Jul 23 00:20:39.506280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:39.506239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:39.906050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:39.906014 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:40.010523 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.010484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:40.010708 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.010662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:40.010708 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.010698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:40.010821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.010733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:40.010821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.010775 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:40.013562 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.013511 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"image-registry-66c98f88f4-c2vjt\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:40.013689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.013677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa4ae49-ef3b-49c5-8267-fcd94a9127fa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-gfnsp\" (UID: \"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:40.013752 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.013729 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/417d7eb2-478f-44cd-aed6-554556aa68d8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxpl4\" (UID: \"417d7eb2-478f-44cd-aed6-554556aa68d8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:40.013752 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.013736 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a40c425e-9a51-48a3-ad58-96eda4dddb66-metrics-tls\") pod \"dns-default-jhz52\" (UID: \"a40c425e-9a51-48a3-ad58-96eda4dddb66\") " pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:40.013949 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.013932 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"image-registry-7f665b65c-w5xjk\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:40.025876 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.025855 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hzp88\"" Jul 23 00:20:40.034811 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.034791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:40.111288 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.111267 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:40.111407 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.111304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:40.111407 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.111364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:40.112413 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.112359 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-service-ca-bundle\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:40.112413 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.112358 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:40.113768 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.113725 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/48d9735a-9a29-47c3-a62c-6626b8f341b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-ndsgc\" (UID: \"48d9735a-9a29-47c3-a62c-6626b8f341b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:40.113924 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.113905 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fca11ac-2e0f-4249-9bfd-060eeae9dbd3-metrics-certs\") pod \"router-default-f6fc97d76-kshb5\" (UID: \"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3\") " pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:40.172574 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.172505 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t6tl2\"" Jul 23 00:20:40.179004 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.178975 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhz52"] Jul 23 00:20:40.180847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.180824 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" Jul 23 00:20:40.181645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.181608 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:40.183571 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:40.183543 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40c425e_9a51_48a3_ad58_96eda4dddb66.slice/crio-79a0e66c5bc5a172b0b41dfeef906d785ede02962b26d59dfd6bd5eff149eee0 WatchSource:0}: Error finding container 79a0e66c5bc5a172b0b41dfeef906d785ede02962b26d59dfd6bd5eff149eee0: Status 404 returned error can't find the container with id 79a0e66c5bc5a172b0b41dfeef906d785ede02962b26d59dfd6bd5eff149eee0 Jul 23 00:20:40.185262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.185240 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8wqt5\"" Jul 23 00:20:40.194230 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.194205 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:40.208117 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.208092 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zhk5t\"" Jul 23 00:20:40.216870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.216850 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" Jul 23 00:20:40.235580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.235455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4ls9t\"" Jul 23 00:20:40.241143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.240983 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:40.244625 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.244587 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" Jul 23 00:20:40.263491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.263386 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wf27l\"" Jul 23 00:20:40.271390 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.271368 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:40.367882 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.367440 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4"] Jul 23 00:20:40.379349 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:40.376855 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d7eb2_478f_44cd_aed6_554556aa68d8.slice/crio-cf8b89d0e0cfaa5bb8f13ad5295d13aa56650da0745f99445aefe8b40fcd8530 WatchSource:0}: Error finding container cf8b89d0e0cfaa5bb8f13ad5295d13aa56650da0745f99445aefe8b40fcd8530: Status 404 returned error can't find the container with id cf8b89d0e0cfaa5bb8f13ad5295d13aa56650da0745f99445aefe8b40fcd8530 Jul 23 00:20:40.406314 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.405907 2585 scope.go:117] "RemoveContainer" containerID="d8cd98aaae0ea7b68286b4adb89bb1be61b3df345c822182bc9e85ca2d88f6d3" Jul 23 00:20:40.412583 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.412515 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f665b65c-w5xjk"] Jul 23 00:20:40.457621 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.447870 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" event={"ID":"417d7eb2-478f-44cd-aed6-554556aa68d8","Type":"ContainerStarted","Data":"cf8b89d0e0cfaa5bb8f13ad5295d13aa56650da0745f99445aefe8b40fcd8530"} Jul 23 00:20:40.457621 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.450125 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhz52" event={"ID":"a40c425e-9a51-48a3-ad58-96eda4dddb66","Type":"ContainerStarted","Data":"79a0e66c5bc5a172b0b41dfeef906d785ede02962b26d59dfd6bd5eff149eee0"} Jul 23 00:20:40.457621 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.452085 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" event={"ID":"75028301-5e65-4c43-8b06-28b6454f9820","Type":"ContainerStarted","Data":"669b2ae3e4be05cdaa0c416e4b6f1258b752f06cb70ff29af4d32044e75f01cf"} Jul 23 00:20:40.497891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.497797 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp"] Jul 23 00:20:40.547156 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.546649 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66c98f88f4-c2vjt"] Jul 23 00:20:40.550002 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:40.549971 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4c4590_1220_4182_b2d7_d07077143423.slice/crio-541bb538da5c6693b6645a9d26221d384400d6eb60c74919ad90276c05218b4f WatchSource:0}: Error finding container 541bb538da5c6693b6645a9d26221d384400d6eb60c74919ad90276c05218b4f: Status 404 returned error can't find the container with id 541bb538da5c6693b6645a9d26221d384400d6eb60c74919ad90276c05218b4f Jul 23 00:20:40.562955 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.562930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f6fc97d76-kshb5"] Jul 23 00:20:40.566508 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:40.565877 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fca11ac_2e0f_4249_9bfd_060eeae9dbd3.slice/crio-2144259a5d416bddc9e8c265704dabb466fa0ade23f28d548a9cdfadbc95c7e1 WatchSource:0}: Error finding container 2144259a5d416bddc9e8c265704dabb466fa0ade23f28d548a9cdfadbc95c7e1: Status 404 returned error can't find the container with id 2144259a5d416bddc9e8c265704dabb466fa0ade23f28d548a9cdfadbc95c7e1 Jul 23 00:20:40.580895 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:40.580750 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc"] Jul 23 00:20:40.585142 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:40.585099 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d9735a_9a29_47c3_a62c_6626b8f341b2.slice/crio-bde983725f92a020dcf236da1e651574c45e25518e5211ef226cdd6c58e2ed1c WatchSource:0}: Error finding container bde983725f92a020dcf236da1e651574c45e25518e5211ef226cdd6c58e2ed1c: Status 404 returned error can't find the container with id bde983725f92a020dcf236da1e651574c45e25518e5211ef226cdd6c58e2ed1c Jul 23 00:20:41.151320 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.151219 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:41.251977 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.251940 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:41.338194 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.338145 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:41.438274 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.438148 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:41.459706 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.459671 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" event={"ID":"ae4c4590-1220-4182-b2d7-d07077143423","Type":"ContainerStarted","Data":"35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2"} Jul 23 00:20:41.459859 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.459735 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" event={"ID":"ae4c4590-1220-4182-b2d7-d07077143423","Type":"ContainerStarted","Data":"541bb538da5c6693b6645a9d26221d384400d6eb60c74919ad90276c05218b4f"} Jul 23 00:20:41.459969 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.459950 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:20:41.463346 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.463324 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:20:41.463446 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.463420 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" event={"ID":"92dd474f-1612-4464-ae50-4a24c828c187","Type":"ContainerStarted","Data":"bd48786e76f870d82c7bec1f8e7cd6489ddf0fe97c14f270248bcfd0fa369d5f"} Jul 23 00:20:41.463947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.463924 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:41.465806 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.465775 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" event={"ID":"48d9735a-9a29-47c3-a62c-6626b8f341b2","Type":"ContainerStarted","Data":"bde983725f92a020dcf236da1e651574c45e25518e5211ef226cdd6c58e2ed1c"} Jul 23 00:20:41.467324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.467303 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f6fc97d76-kshb5" event={"ID":"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3","Type":"ContainerStarted","Data":"7b2b66186122e442b11694b8562fbb406ef64b79568264ad7c03b83fc4610acc"} Jul 23 00:20:41.467442 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.467345 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f6fc97d76-kshb5" event={"ID":"5fca11ac-2e0f-4249-9bfd-060eeae9dbd3","Type":"ContainerStarted","Data":"2144259a5d416bddc9e8c265704dabb466fa0ade23f28d548a9cdfadbc95c7e1"} Jul 23 00:20:41.470192 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.470146 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" Jul 23 00:20:41.471179 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.471113 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" event={"ID":"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa","Type":"ContainerStarted","Data":"103cb461d5d3702546ecc0ae071b78376920598d6a47f6e86c93e8f62fcb999e"} Jul 23 00:20:41.474990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.474454 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" event={"ID":"75028301-5e65-4c43-8b06-28b6454f9820","Type":"ContainerStarted","Data":"07b5f010a2a5c47662c73933971842e1613a409feb746f44876ce4c84ad84b4f"} Jul 23 00:20:41.474990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.474950 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:20:41.478624 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.478583 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" podStartSLOduration=46.478559163 podStartE2EDuration="46.478559163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:41.477558946 +0000 UTC m=+70.774253933" watchObservedRunningTime="2026-07-23 00:20:41.478559163 +0000 UTC m=+70.775254130" Jul 23 00:20:41.496671 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.496602 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" podStartSLOduration=70.496584602 podStartE2EDuration="1m10.496584602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:41.494960135 +0000 UTC m=+70.791655097" watchObservedRunningTime="2026-07-23 00:20:41.496584602 +0000 UTC m=+70.793279570" Jul 23 00:20:41.515846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.515801 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-f6fc97d76-kshb5" podStartSLOduration=46.515785176 podStartE2EDuration="46.515785176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:41.514963707 +0000 UTC m=+70.811658720" watchObservedRunningTime="2026-07-23 00:20:41.515785176 +0000 UTC m=+70.812480125" Jul 23 00:20:41.616698 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.616626 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:41.673510 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.673477 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:41.729325 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.729242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:41.729325 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.729301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:41.730288 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.730263 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a1deeea-5ee1-46da-8634-c754a6a24cc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:41.733458 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.733433 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a1deeea-5ee1-46da-8634-c754a6a24cc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vjbl8\" (UID: \"3a1deeea-5ee1-46da-8634-c754a6a24cc0\") " pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:41.757755 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.757715 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t68f8\"" Jul 23 00:20:41.766570 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:41.766548 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vjbl8" Jul 23 00:20:42.271736 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:42.271603 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:42.274802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:42.274563 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:42.292290 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:42.292237 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-wvjzl" podStartSLOduration=33.789170995 podStartE2EDuration="47.292219892s" totalImagesPullingTime="13.503048897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:41.534114968 +0000 UTC m=+70.830809934" watchObservedRunningTime="2026-07-23 00:20:42.292219892 +0000 UTC m=+71.588914862" Jul 23 00:20:42.479232 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:42.479175 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:42.480106 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:42.480084 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f6fc97d76-kshb5" Jul 23 00:20:42.830545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:42.830436 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:43.033836 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:43.033791 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:43.134775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:43.134649 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:43.235884 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:43.235823 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.247974 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:44.247935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.298056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:44.298023 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.964854 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:44.964775 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vjbl8"] Jul 23 00:20:45.211162 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.211115 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:45.311980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.311895 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:45.412692 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.412658 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:45.492658 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.492622 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-spxkd" event={"ID":"cd3be28b-aa8d-4a5a-b230-dbedffd5543c","Type":"ContainerStarted","Data":"d55d959ef82e25d699eac64154a0769c2fff7da768f22af6b1fbb1393ea5554f"} Jul 23 00:20:45.494170 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.494144 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" event={"ID":"417d7eb2-478f-44cd-aed6-554556aa68d8","Type":"ContainerStarted","Data":"417d89cc07cd3db74fced4671aea7b6e29b5cb6fe3b0fadd143b425842c66f0d"} Jul 23 00:20:45.496018 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.495995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhz52" event={"ID":"a40c425e-9a51-48a3-ad58-96eda4dddb66","Type":"ContainerStarted","Data":"783642a4e77a81214a279bdeaf61ba45b03527297be11142eb3b88dc8de33c9e"} Jul 23 00:20:45.496133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.496031 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhz52" event={"ID":"a40c425e-9a51-48a3-ad58-96eda4dddb66","Type":"ContainerStarted","Data":"7269cdfa8b4da0e2ad2e74095da63a2583e8df883ce7f327e78c428569b968e7"} Jul 23 00:20:45.496133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.496075 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:45.497499 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.497475 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" event={"ID":"48d9735a-9a29-47c3-a62c-6626b8f341b2","Type":"ContainerStarted","Data":"1186c6c3c465337b166787e2dd79f8414a3307d64a00b15d2deec9a9022e4c93"} Jul 23 00:20:45.499194 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.499173 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" event={"ID":"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa","Type":"ContainerStarted","Data":"b2fb846c896ad17a98cf03c00f730695c9646994ddbcde7e08e1ef8ce08a6bc4"} Jul 23 00:20:45.499309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.499217 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" event={"ID":"6aa4ae49-ef3b-49c5-8267-fcd94a9127fa","Type":"ContainerStarted","Data":"e3ba05f506fb92e1b14cc08225a1c53351fdd39bc4274238559595bb294474a2"} Jul 23 00:20:45.500272 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.500251 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vjbl8" event={"ID":"3a1deeea-5ee1-46da-8634-c754a6a24cc0","Type":"ContainerStarted","Data":"956c3b14f66bbd8fbe3adfce035e782e52c7c67591d3171d095c04318da37000"} Jul 23 00:20:45.506448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.506404 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-spxkd" podStartSLOduration=67.049491745 podStartE2EDuration="1m14.506387323s" totalImagesPullingTime="7.456895578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:45.505673619 +0000 UTC m=+74.802368586" watchObservedRunningTime="2026-07-23 00:20:45.506387323 +0000 UTC m=+74.803082292" Jul 23 00:20:45.510128 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.510104 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:45.520969 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.520928 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-ndsgc" podStartSLOduration=46.295316634 podStartE2EDuration="50.520916604s" totalImagesPullingTime="4.22559997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:45.519039005 +0000 UTC m=+74.815733993" watchObservedRunningTime="2026-07-23 00:20:45.520916604 +0000 UTC m=+74.817611572" Jul 23 00:20:45.535053 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.535008 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxpl4" podStartSLOduration=46.102967612 podStartE2EDuration="50.534997038s" totalImagesPullingTime="4.432029426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:45.534599823 +0000 UTC m=+74.831294811" watchObservedRunningTime="2026-07-23 00:20:45.534997038 +0000 UTC m=+74.831692036" Jul 23 00:20:45.549285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.549246 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-gfnsp" podStartSLOduration=46.308127917 podStartE2EDuration="50.549234909s" totalImagesPullingTime="4.241106992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:45.548273615 +0000 UTC m=+74.844968582" watchObservedRunningTime="2026-07-23 00:20:45.549234909 +0000 UTC m=+74.845929877" Jul 23 00:20:45.562874 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:45.562795 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jhz52" podStartSLOduration=32.932193624 podStartE2EDuration="37.562782946s" totalImagesPullingTime="4.630589322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:45.561241637 +0000 UTC m=+74.857936616" watchObservedRunningTime="2026-07-23 00:20:45.562782946 +0000 UTC m=+74.859477913" Jul 23 00:20:47.216877 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.216837 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6"] Jul 23 00:20:47.257790 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.257751 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6"] Jul 23 00:20:47.257790 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.257795 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66c98f88f4-c2vjt"] Jul 23 00:20:47.258051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.257878 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:47.259927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.259901 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ntgx7\"" Jul 23 00:20:47.260067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.259980 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 00:20:47.316289 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.316247 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:47.319054 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.318759 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-2x84q"] Jul 23 00:20:47.339039 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.338891 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-2x84q"] Jul 23 00:20:47.339039 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.338987 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:20:47.341486 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.341436 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6h8s2\"" Jul 23 00:20:47.341592 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.341553 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 00:20:47.341648 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.341436 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 00:20:47.382610 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.382581 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ed447874-471c-41d7-ae97-8b505d19dcd9-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mhwz6\" (UID: \"ed447874-471c-41d7-ae97-8b505d19dcd9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:47.443883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.443850 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:47.483717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.483683 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ed447874-471c-41d7-ae97-8b505d19dcd9-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mhwz6\" (UID: \"ed447874-471c-41d7-ae97-8b505d19dcd9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:47.483863 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.483818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nhm\" (UniqueName: \"kubernetes.io/projected/1a029abf-df82-48a2-aea2-531b0b24d44b-kube-api-access-l4nhm\") pod \"downloads-5bc67dcd46-2x84q\" (UID: \"1a029abf-df82-48a2-aea2-531b0b24d44b\") " pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:20:47.486971 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.486941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ed447874-471c-41d7-ae97-8b505d19dcd9-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mhwz6\" (UID: \"ed447874-471c-41d7-ae97-8b505d19dcd9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:47.519513 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.519458 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:47.571646 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.571620 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:47.585073 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.585045 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nhm\" (UniqueName: \"kubernetes.io/projected/1a029abf-df82-48a2-aea2-531b0b24d44b-kube-api-access-l4nhm\") pod \"downloads-5bc67dcd46-2x84q\" (UID: \"1a029abf-df82-48a2-aea2-531b0b24d44b\") " pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:20:47.591782 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.591763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nhm\" (UniqueName: \"kubernetes.io/projected/1a029abf-df82-48a2-aea2-531b0b24d44b-kube-api-access-l4nhm\") pod \"downloads-5bc67dcd46-2x84q\" (UID: \"1a029abf-df82-48a2-aea2-531b0b24d44b\") " pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:20:47.614944 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.614910 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:47.649847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.649818 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:20:47.716644 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:47.716404 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6"] Jul 23 00:20:49.567262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:49.567230 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:49.620587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:49.620554 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:49.828813 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:49.828743 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded447874_471c_41d7_ae97_8b505d19dcd9.slice/crio-37e935ee5f28d6de1179eed73f7dd7e0f6c8019000ccd8545a37718e3793a145 WatchSource:0}: Error finding container 37e935ee5f28d6de1179eed73f7dd7e0f6c8019000ccd8545a37718e3793a145: Status 404 returned error can't find the container with id 37e935ee5f28d6de1179eed73f7dd7e0f6c8019000ccd8545a37718e3793a145 Jul 23 00:20:49.863555 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:49.863522 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:49.914108 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:49.914083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:50.467952 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.467920 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c88fd7fc-5mmld"] Jul 23 00:20:50.471390 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.471326 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:50.500290 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.500267 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c88fd7fc-5mmld"] Jul 23 00:20:50.500434 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.500370 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.503316 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.503293 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 00:20:50.503424 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.503293 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 00:20:50.503424 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.503388 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 00:20:50.503424 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.503293 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 00:20:50.503614 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.503295 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 00:20:50.503718 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.503693 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjlcf\"" Jul 23 00:20:50.510342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.510319 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-oauth-config\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.510444 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.510361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv77k\" (UniqueName: \"kubernetes.io/projected/0c2574a9-6681-4810-814e-18238be3a296-kube-api-access-qv77k\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.510444 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.510397 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-serving-cert\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.510444 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.510426 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-oauth-serving-cert\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.510599 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.510487 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-service-ca\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.510599 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.510579 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-console-config\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.522514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.522483 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" event={"ID":"ed447874-471c-41d7-ae97-8b505d19dcd9","Type":"ContainerStarted","Data":"37e935ee5f28d6de1179eed73f7dd7e0f6c8019000ccd8545a37718e3793a145"} Jul 23 00:20:50.546396 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.546354 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:50.611059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.611025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-service-ca\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.611511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.611084 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-console-config\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.611511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.611145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-oauth-config\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.611511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.611173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv77k\" (UniqueName: \"kubernetes.io/projected/0c2574a9-6681-4810-814e-18238be3a296-kube-api-access-qv77k\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.611511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.611204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-serving-cert\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.611511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.611236 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-oauth-serving-cert\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.612135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.612100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-console-config\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.612231 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.612100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-service-ca\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.612231 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.612208 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-oauth-serving-cert\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.614398 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.614371 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-oauth-config\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.614514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.614452 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-serving-cert\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.618395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.618377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv77k\" (UniqueName: \"kubernetes.io/projected/0c2574a9-6681-4810-814e-18238be3a296-kube-api-access-qv77k\") pod \"console-59c88fd7fc-5mmld\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.812588 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.812551 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:20:50.873422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.873393 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-2x84q"] Jul 23 00:20:50.874506 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:50.874459 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a029abf_df82_48a2_aea2_531b0b24d44b.slice/crio-ba9b762091241b79cbb29878bda239323801ff6813ccf0bb611a4643a4539eb0 WatchSource:0}: Error finding container ba9b762091241b79cbb29878bda239323801ff6813ccf0bb611a4643a4539eb0: Status 404 returned error can't find the container with id ba9b762091241b79cbb29878bda239323801ff6813ccf0bb611a4643a4539eb0 Jul 23 00:20:50.960067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:50.960034 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c88fd7fc-5mmld"] Jul 23 00:20:51.096225 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:20:51.096145 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2574a9_6681_4810_814e_18238be3a296.slice/crio-84356694480d6f0cb809143019be94d0f589335727aa934992bdbbbeeeaccda3 WatchSource:0}: Error finding container 84356694480d6f0cb809143019be94d0f589335727aa934992bdbbbeeeaccda3: Status 404 returned error can't find the container with id 84356694480d6f0cb809143019be94d0f589335727aa934992bdbbbeeeaccda3 Jul 23 00:20:51.533850 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:51.533811 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vjbl8" event={"ID":"3a1deeea-5ee1-46da-8634-c754a6a24cc0","Type":"ContainerStarted","Data":"2fd7294a2fe7c35c5530ca9b7523bbaccbe6616302fb5aa83cd1768e3a366c40"} Jul 23 00:20:51.535109 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:51.535082 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-2x84q" event={"ID":"1a029abf-df82-48a2-aea2-531b0b24d44b","Type":"ContainerStarted","Data":"ba9b762091241b79cbb29878bda239323801ff6813ccf0bb611a4643a4539eb0"} Jul 23 00:20:51.536479 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:51.536435 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c88fd7fc-5mmld" event={"ID":"0c2574a9-6681-4810-814e-18238be3a296","Type":"ContainerStarted","Data":"84356694480d6f0cb809143019be94d0f589335727aa934992bdbbbeeeaccda3"} Jul 23 00:20:51.807198 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:51.807112 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:51.969937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:51.969893 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:52.070693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:52.070591 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:52.546826 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:52.546787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" event={"ID":"ed447874-471c-41d7-ae97-8b505d19dcd9","Type":"ContainerStarted","Data":"5b28ba93c9713143caeb1a2f903cf64274da36ff52dc1c9d2f240103c78f6d57"} Jul 23 00:20:52.547327 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:52.547262 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:52.553788 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:52.553734 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" Jul 23 00:20:52.562186 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:52.562062 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mhwz6" podStartSLOduration=3.250114639 podStartE2EDuration="5.562043947s" totalImagesPullingTime="2.311929308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:52.561076389 +0000 UTC m=+81.857771372" watchObservedRunningTime="2026-07-23 00:20:52.562043947 +0000 UTC m=+81.858738910" Jul 23 00:20:53.484192 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:53.484017 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:53.548015 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:53.547971 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:53.814654 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:53.814564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:54.410678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:54.410646 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:55.508061 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:55.508031 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jhz52" Jul 23 00:20:57.264972 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:57.264935 2585 patch_prober.go:28] interesting pod/image-registry-66c98f88f4-c2vjt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 00:20:57.265534 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:57.264997 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" podUID="ae4c4590-1220-4182-b2d7-d07077143423" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:20:58.580282 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:58.580240 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vjbl8" event={"ID":"3a1deeea-5ee1-46da-8634-c754a6a24cc0","Type":"ContainerStarted","Data":"ccfe2c4ca1cda9991e70ef01385ceef59d40eb226217e40887a5c87fa4647732"} Jul 23 00:20:58.581955 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:58.581924 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c88fd7fc-5mmld" event={"ID":"0c2574a9-6681-4810-814e-18238be3a296","Type":"ContainerStarted","Data":"b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f"} Jul 23 00:20:58.597389 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:58.597163 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59c88fd7fc-5mmld" podStartSLOduration=1.9311203190000001 podStartE2EDuration="8.597146767s" totalImagesPullingTime="6.666026448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:58.597111749 +0000 UTC m=+87.893806747" watchObservedRunningTime="2026-07-23 00:20:58.597146767 +0000 UTC m=+87.893841736" Jul 23 00:20:59.010448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:59.010405 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:59.110361 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:59.110247 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:59.657066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:59.657023 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:59.758534 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:20:59.758346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:00.201441 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:00.201394 2585 patch_prober.go:28] interesting pod/image-registry-7f665b65c-w5xjk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 00:21:00.201738 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:00.201702 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" podUID="75028301-5e65-4c43-8b06-28b6454f9820" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:21:00.812923 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:00.812887 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:21:00.813352 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:00.812958 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:21:00.818573 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:00.818546 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:21:01.603481 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:01.603430 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:21:02.493784 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:02.493716 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:02.605666 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:02.605629 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vjbl8" event={"ID":"3a1deeea-5ee1-46da-8634-c754a6a24cc0","Type":"ContainerStarted","Data":"1442b99872c2d2bf57eb47f610e6f65051a749d0d8a6742a3c584546d3cf3ece"} Jul 23 00:21:02.622262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:02.622168 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vjbl8" podStartSLOduration=20.475035312 podStartE2EDuration="37.622148066s" totalImagesPullingTime="17.147112754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:20:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:02.621092217 +0000 UTC m=+91.917787189" watchObservedRunningTime="2026-07-23 00:21:02.622148066 +0000 UTC m=+91.918843035" Jul 23 00:21:03.254174 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:03.254129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:03.354720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:03.354675 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:03.488813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:03.488784 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:21:03.633676 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:03.633570 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:03.734252 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:03.734202 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.548792 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.548699 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.758483 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.757999 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.828594 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.828517 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n9xvd"] Jul 23 00:21:04.831968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.831946 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.834062 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.833876 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 00:21:04.834062 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.833933 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 00:21:04.834427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.834407 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 00:21:04.834554 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.834485 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 00:21:04.834554 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.834496 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mjp2b\"" Jul 23 00:21:04.844956 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.844816 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.925363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.925200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.944455 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-root\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944455 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944416 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944455 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-tls\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944519 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944548 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv28p\" (UniqueName: \"kubernetes.io/projected/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-kube-api-access-dv28p\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944594 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-metrics-client-ca\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944631 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-wtmp\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944672 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-sys\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:04.944852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:04.944692 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-textfile\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045193 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv28p\" (UniqueName: \"kubernetes.io/projected/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-kube-api-access-dv28p\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-metrics-client-ca\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045279 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-wtmp\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-sys\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-textfile\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-root\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.045434 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-tls\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.046145 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:21:05.045614 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 00:21:05.046770 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:21:05.046164 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-tls podName:5a464fcd-5763-44fe-8f3b-c7a7cf8b5709 nodeName:}" failed. No retries permitted until 2026-07-23 00:21:05.545665463 +0000 UTC m=+94.842360414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-tls") pod "node-exporter-n9xvd" (UID: "5a464fcd-5763-44fe-8f3b-c7a7cf8b5709") : secret "node-exporter-tls" not found Jul 23 00:21:05.046770 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.046373 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-sys\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.047379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.047031 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-wtmp\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.047379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.047090 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-root\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.047379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.047132 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-metrics-client-ca\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.047379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.047329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-textfile\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.047659 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.047583 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.053610 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.053564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.057690 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.057649 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv28p\" (UniqueName: \"kubernetes.io/projected/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-kube-api-access-dv28p\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.086425 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.086255 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:05.186596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.186557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:05.285125 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.285085 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:05.384720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.384625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:05.447442 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.447403 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c88fd7fc-5mmld"] Jul 23 00:21:05.550727 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.550689 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-tls\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.553141 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.553113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a464fcd-5763-44fe-8f3b-c7a7cf8b5709-node-exporter-tls\") pod \"node-exporter-n9xvd\" (UID: \"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709\") " pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.744289 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.744252 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9xvd" Jul 23 00:21:05.793904 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.793811 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:05.873634 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.873583 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:05.994611 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:05.994485 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:06.099521 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:06.099482 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:06.457928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:06.457825 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:06.558778 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:06.558740 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:07.263306 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:07.263278 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:21:07.458934 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:07.458900 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:07.765063 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:07.765030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:07.834015 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:07.833981 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:07.937532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:07.937492 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:08.072082 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:08.071996 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:08.272867 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:08.272837 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:08.320239 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:08.320198 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:09.437113 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:09.437083 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gdqcl" Jul 23 00:21:09.589324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:09.589277 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:09.644728 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:09.644692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:09.865589 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:09.865556 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:10.110369 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:10.110340 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:10.179322 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:21:10.179293 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a464fcd_5763_44fe_8f3b_c7a7cf8b5709.slice/crio-bc30870dc900c4c1e51da01b6fbaabedd062bf40a52dfa59c120e5fdd40a965f WatchSource:0}: Error finding container bc30870dc900c4c1e51da01b6fbaabedd062bf40a52dfa59c120e5fdd40a965f: Status 404 returned error can't find the container with id bc30870dc900c4c1e51da01b6fbaabedd062bf40a52dfa59c120e5fdd40a965f Jul 23 00:21:10.187399 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:10.187364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:10.551969 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:10.551928 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:10.645997 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:10.645971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9xvd" event={"ID":"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709","Type":"ContainerStarted","Data":"bc30870dc900c4c1e51da01b6fbaabedd062bf40a52dfa59c120e5fdd40a965f"} Jul 23 00:21:10.647309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:10.647289 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-2x84q" event={"ID":"1a029abf-df82-48a2-aea2-531b0b24d44b","Type":"ContainerStarted","Data":"bf03418a8886e4a552b32bac1a0b7b5a2cd782ee87210becedb22250bb0cc936"} Jul 23 00:21:10.652669 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:10.652639 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:11.036574 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:11.036388 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:11.656157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:11.656130 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:21:11.660678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:11.660653 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-2x84q" Jul 23 00:21:11.675645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:11.675603 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-2x84q" podStartSLOduration=4.997328737 podStartE2EDuration="24.675589187s" totalImagesPullingTime="19.67826045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:11.673999529 +0000 UTC m=+100.970694498" watchObservedRunningTime="2026-07-23 00:21:11.675589187 +0000 UTC m=+100.972284155" Jul 23 00:21:12.120269 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.120172 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:12.220891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.220856 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:12.283884 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.283845 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" podUID="ae4c4590-1220-4182-b2d7-d07077143423" containerName="registry" containerID="cri-o://35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2" gracePeriod=30 Jul 23 00:21:12.438883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.438847 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:12.515393 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.515356 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:12.610151 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.610119 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:21:12.662229 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.662185 2585 generic.go:356] "Generic (PLEG): container finished" podID="ae4c4590-1220-4182-b2d7-d07077143423" containerID="35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2" exitCode=0 Jul 23 00:21:12.662694 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.662247 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" Jul 23 00:21:12.662694 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.662315 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" event={"ID":"ae4c4590-1220-4182-b2d7-d07077143423","Type":"ContainerDied","Data":"35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2"} Jul 23 00:21:12.662694 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.662365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c98f88f4-c2vjt" event={"ID":"ae4c4590-1220-4182-b2d7-d07077143423","Type":"ContainerDied","Data":"541bb538da5c6693b6645a9d26221d384400d6eb60c74919ad90276c05218b4f"} Jul 23 00:21:12.662694 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.662396 2585 scope.go:117] "RemoveContainer" containerID="35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2" Jul 23 00:21:12.665041 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.664946 2585 generic.go:356] "Generic (PLEG): container finished" podID="5a464fcd-5763-44fe-8f3b-c7a7cf8b5709" containerID="7e20172327c505318dd1e47e7a951de0e1ac04d4985e5a2f6bc17816d2922676" exitCode=0 Jul 23 00:21:12.665747 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.665725 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9xvd" event={"ID":"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709","Type":"ContainerDied","Data":"7e20172327c505318dd1e47e7a951de0e1ac04d4985e5a2f6bc17816d2922676"} Jul 23 00:21:12.673732 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.673686 2585 scope.go:117] "RemoveContainer" containerID="35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2" Jul 23 00:21:12.674052 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:21:12.674023 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2\": container with ID starting with 35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2 not found: ID does not exist" containerID="35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2" Jul 23 00:21:12.674316 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.674063 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2"} err="failed to get container status \"35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2\": rpc error: code = NotFound desc = could not find container \"35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2\": container with ID starting with 35f24c83d42d7d4e16c46bf825edb8f22fc5b3524bb1ffdaf0911d55b48539c2 not found: ID does not exist" Jul 23 00:21:12.717775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.717746 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-installation-pull-secrets\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.717912 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.717788 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.717912 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.717856 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-trusted-ca\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.718023 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.717910 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb2tk\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-kube-api-access-kb2tk\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.718023 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.717954 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4c4590-1220-4182-b2d7-d07077143423-ca-trust-extracted\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.718023 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.717990 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-image-registry-private-configuration\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.718180 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.718015 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-registry-certificates\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.718180 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.718059 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-bound-sa-token\") pod \"ae4c4590-1220-4182-b2d7-d07077143423\" (UID: \"ae4c4590-1220-4182-b2d7-d07077143423\") " Jul 23 00:21:12.719041 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.718450 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:12.719041 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.718777 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:12.720870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.719686 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-trusted-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.720870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.719712 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4c4590-1220-4182-b2d7-d07077143423-registry-certificates\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.724749 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.724718 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:12.724860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.724834 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:12.724943 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.724924 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:12.730378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.730309 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4c4590-1220-4182-b2d7-d07077143423-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:21:12.732379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.732332 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-kube-api-access-kb2tk" (OuterVolumeSpecName: "kube-api-access-kb2tk") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "kube-api-access-kb2tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:12.732839 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.732774 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ae4c4590-1220-4182-b2d7-d07077143423" (UID: "ae4c4590-1220-4182-b2d7-d07077143423"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:12.820774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.820740 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-installation-pull-secrets\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.820774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.820776 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-registry-tls\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.821005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.820791 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb2tk\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-kube-api-access-kb2tk\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.821005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.820805 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4c4590-1220-4182-b2d7-d07077143423-ca-trust-extracted\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.821005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.820824 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4c4590-1220-4182-b2d7-d07077143423-image-registry-private-configuration\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.821005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.820840 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c4590-1220-4182-b2d7-d07077143423-bound-sa-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:12.992277 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.992251 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66c98f88f4-c2vjt"] Jul 23 00:21:12.998898 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:12.998872 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66c98f88f4-c2vjt"] Jul 23 00:21:13.413125 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.413085 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae4c4590-1220-4182-b2d7-d07077143423" path="/var/lib/kubelet/pods/ae4c4590-1220-4182-b2d7-d07077143423/volumes" Jul 23 00:21:13.504701 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.504673 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f665b65c-w5xjk"] Jul 23 00:21:13.595737 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.595701 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:13.674487 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.674386 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9xvd" event={"ID":"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709","Type":"ContainerStarted","Data":"9dcf68857530c4726354150af74ff5895018f56eff9798eab22b9d876ad1fabc"} Jul 23 00:21:13.674937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.674449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9xvd" event={"ID":"5a464fcd-5763-44fe-8f3b-c7a7cf8b5709","Type":"ContainerStarted","Data":"47006e06b1d901e1e804f057149d94a5ab0a74df352eb58863f0031688ead9aa"} Jul 23 00:21:13.699337 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.699308 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:13.699586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:13.699526 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n9xvd" podStartSLOduration=8.223712004 podStartE2EDuration="9.699506669s" totalImagesPullingTime="1.361655639s" totalInitContainerRuntime="114.139026ms" isStatefulPod=true podCreationTimestamp="2026-07-23 00:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:13.698322631 +0000 UTC m=+102.995017621" watchObservedRunningTime="2026-07-23 00:21:13.699506669 +0000 UTC m=+102.996201628" Jul 23 00:21:14.257568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:14.257532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:14.655148 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:14.655067 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:17.341001 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:17.340960 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:17.651082 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:17.651001 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:17.756254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:17.756215 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:17.851803 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:17.851766 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:18.590556 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:18.590518 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:18.697080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:18.697045 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:20.629236 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:20.629200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:20.727769 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:20.727733 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:27.606322 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:27.606283 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:29.254105 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:29.254066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:29.353631 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:29.353598 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:29.716029 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:29.715951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:29.768287 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:29.768249 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:29.821010 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:29.820978 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:30.471314 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.471278 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59c88fd7fc-5mmld" podUID="0c2574a9-6681-4810-814e-18238be3a296" containerName="console" containerID="cri-o://b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f" gracePeriod=15 Jul 23 00:21:30.744563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.744535 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:21:30.760991 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.760963 2585 generic.go:356] "Generic (PLEG): container finished" podID="0c2574a9-6681-4810-814e-18238be3a296" containerID="b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f" exitCode=0 Jul 23 00:21:30.761136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.761022 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c88fd7fc-5mmld" Jul 23 00:21:30.761136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.761041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c88fd7fc-5mmld" event={"ID":"0c2574a9-6681-4810-814e-18238be3a296","Type":"ContainerDied","Data":"b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f"} Jul 23 00:21:30.761136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.761103 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c88fd7fc-5mmld" event={"ID":"0c2574a9-6681-4810-814e-18238be3a296","Type":"ContainerDied","Data":"84356694480d6f0cb809143019be94d0f589335727aa934992bdbbbeeeaccda3"} Jul 23 00:21:30.761136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.761130 2585 scope.go:117] "RemoveContainer" containerID="b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f" Jul 23 00:21:30.770522 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.770500 2585 scope.go:117] "RemoveContainer" containerID="b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f" Jul 23 00:21:30.770768 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:21:30.770750 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f\": container with ID starting with b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f not found: ID does not exist" containerID="b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f" Jul 23 00:21:30.770820 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.770777 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f"} err="failed to get container status \"b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f\": rpc error: code = NotFound desc = could not find container \"b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f\": container with ID starting with b1f8df79d6413926416e2b1e1f6d35f94864e1c99b30213efb897dda5dceb93f not found: ID does not exist" Jul 23 00:21:30.877924 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.877895 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-oauth-config\") pod \"0c2574a9-6681-4810-814e-18238be3a296\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " Jul 23 00:21:30.878091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.877947 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-serving-cert\") pod \"0c2574a9-6681-4810-814e-18238be3a296\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " Jul 23 00:21:30.878091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.877991 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-service-ca\") pod \"0c2574a9-6681-4810-814e-18238be3a296\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " Jul 23 00:21:30.878091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.878011 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-oauth-serving-cert\") pod \"0c2574a9-6681-4810-814e-18238be3a296\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " Jul 23 00:21:30.878091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.878035 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-console-config\") pod \"0c2574a9-6681-4810-814e-18238be3a296\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " Jul 23 00:21:30.878091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.878071 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv77k\" (UniqueName: \"kubernetes.io/projected/0c2574a9-6681-4810-814e-18238be3a296-kube-api-access-qv77k\") pod \"0c2574a9-6681-4810-814e-18238be3a296\" (UID: \"0c2574a9-6681-4810-814e-18238be3a296\") " Jul 23 00:21:30.878420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.878398 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0c2574a9-6681-4810-814e-18238be3a296" (UID: "0c2574a9-6681-4810-814e-18238be3a296"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:30.878420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.878410 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-service-ca" (OuterVolumeSpecName: "service-ca") pod "0c2574a9-6681-4810-814e-18238be3a296" (UID: "0c2574a9-6681-4810-814e-18238be3a296"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:30.878627 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.878607 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-console-config" (OuterVolumeSpecName: "console-config") pod "0c2574a9-6681-4810-814e-18238be3a296" (UID: "0c2574a9-6681-4810-814e-18238be3a296"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:30.880162 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.880134 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0c2574a9-6681-4810-814e-18238be3a296" (UID: "0c2574a9-6681-4810-814e-18238be3a296"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:30.880263 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.880233 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2574a9-6681-4810-814e-18238be3a296-kube-api-access-qv77k" (OuterVolumeSpecName: "kube-api-access-qv77k") pod "0c2574a9-6681-4810-814e-18238be3a296" (UID: "0c2574a9-6681-4810-814e-18238be3a296"). InnerVolumeSpecName "kube-api-access-qv77k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:30.880344 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.880323 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0c2574a9-6681-4810-814e-18238be3a296" (UID: "0c2574a9-6681-4810-814e-18238be3a296"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:30.978889 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.978857 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-console-config\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:30.978889 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.978884 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv77k\" (UniqueName: \"kubernetes.io/projected/0c2574a9-6681-4810-814e-18238be3a296-kube-api-access-qv77k\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:30.978889 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.978895 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-oauth-config\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:30.979077 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.978907 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c2574a9-6681-4810-814e-18238be3a296-console-serving-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:30.979077 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.978920 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:30.979077 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:30.978928 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c2574a9-6681-4810-814e-18238be3a296-oauth-serving-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:31.080197 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:31.080154 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c88fd7fc-5mmld"] Jul 23 00:21:31.082325 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:31.082299 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59c88fd7fc-5mmld"] Jul 23 00:21:31.410234 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:31.410157 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2574a9-6681-4810-814e-18238be3a296" path="/var/lib/kubelet/pods/0c2574a9-6681-4810-814e-18238be3a296/volumes" Jul 23 00:21:36.730214 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:36.730176 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:36.830187 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:36.830143 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:36.894486 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:36.894430 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:38.530413 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.530378 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" podUID="75028301-5e65-4c43-8b06-28b6454f9820" containerName="registry" containerID="cri-o://07b5f010a2a5c47662c73933971842e1613a409feb746f44876ce4c84ad84b4f" gracePeriod=30 Jul 23 00:21:38.791043 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.790983 2585 generic.go:356] "Generic (PLEG): container finished" podID="75028301-5e65-4c43-8b06-28b6454f9820" containerID="07b5f010a2a5c47662c73933971842e1613a409feb746f44876ce4c84ad84b4f" exitCode=0 Jul 23 00:21:38.791165 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.791055 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" event={"ID":"75028301-5e65-4c43-8b06-28b6454f9820","Type":"ContainerDied","Data":"07b5f010a2a5c47662c73933971842e1613a409feb746f44876ce4c84ad84b4f"} Jul 23 00:21:38.791165 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.791111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" event={"ID":"75028301-5e65-4c43-8b06-28b6454f9820","Type":"ContainerDied","Data":"669b2ae3e4be05cdaa0c416e4b6f1258b752f06cb70ff29af4d32044e75f01cf"} Jul 23 00:21:38.791165 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.791131 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="669b2ae3e4be05cdaa0c416e4b6f1258b752f06cb70ff29af4d32044e75f01cf" Jul 23 00:21:38.804678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.804655 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:21:38.945948 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.945913 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-bound-sa-token\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.945948 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.945945 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.945969 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-trusted-ca\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.945992 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf7mm\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-kube-api-access-hf7mm\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.946037 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-installation-pull-secrets\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.946087 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75028301-5e65-4c43-8b06-28b6454f9820-ca-trust-extracted\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.946115 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-registry-certificates\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946382 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.946159 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-image-registry-private-configuration\") pod \"75028301-5e65-4c43-8b06-28b6454f9820\" (UID: \"75028301-5e65-4c43-8b06-28b6454f9820\") " Jul 23 00:21:38.946582 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.946534 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:38.946771 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.946605 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:21:38.948546 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.948506 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:38.948546 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.948520 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-kube-api-access-hf7mm" (OuterVolumeSpecName: "kube-api-access-hf7mm") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "kube-api-access-hf7mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:38.948844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.948811 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:38.948954 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.948933 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:38.949013 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.948999 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:38.954773 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:38.954747 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75028301-5e65-4c43-8b06-28b6454f9820-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "75028301-5e65-4c43-8b06-28b6454f9820" (UID: "75028301-5e65-4c43-8b06-28b6454f9820"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047644 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75028301-5e65-4c43-8b06-28b6454f9820-ca-trust-extracted\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047672 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-registry-certificates\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047683 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-image-registry-private-configuration\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047693 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-bound-sa-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047702 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-registry-tls\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047710 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75028301-5e65-4c43-8b06-28b6454f9820-trusted-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047717 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf7mm\" (UniqueName: \"kubernetes.io/projected/75028301-5e65-4c43-8b06-28b6454f9820-kube-api-access-hf7mm\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.047722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.047726 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75028301-5e65-4c43-8b06-28b6454f9820-installation-pull-secrets\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:21:39.794786 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.794756 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f665b65c-w5xjk" Jul 23 00:21:39.809364 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.809337 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f665b65c-w5xjk"] Jul 23 00:21:39.812937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:39.812913 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f665b65c-w5xjk"] Jul 23 00:21:41.051345 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:41.051301 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:41.410690 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:41.410620 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75028301-5e65-4c43-8b06-28b6454f9820" path="/var/lib/kubelet/pods/75028301-5e65-4c43-8b06-28b6454f9820/volumes" Jul 23 00:21:45.819724 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:45.819688 2585 generic.go:356] "Generic (PLEG): container finished" podID="578cdcb0-4740-4a05-9466-3273fd0e990b" containerID="5d9c3a61fff6038f5658f0a346d5941943a146e5066d699db947db1259883795" exitCode=0 Jul 23 00:21:45.820124 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:45.819762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8kf92" event={"ID":"578cdcb0-4740-4a05-9466-3273fd0e990b","Type":"ContainerDied","Data":"5d9c3a61fff6038f5658f0a346d5941943a146e5066d699db947db1259883795"} Jul 23 00:21:45.820168 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:45.820141 2585 scope.go:117] "RemoveContainer" containerID="5d9c3a61fff6038f5658f0a346d5941943a146e5066d699db947db1259883795" Jul 23 00:21:46.826137 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:46.826100 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8kf92" event={"ID":"578cdcb0-4740-4a05-9466-3273fd0e990b","Type":"ContainerStarted","Data":"ad5ecc04e6d625f42c7fb8523d16827f30e6507447e9e48bf132082ad93d7874"} Jul 23 00:21:48.518508 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:48.518450 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" podUID="98d0690d-adb1-4eee-8d63-d9608764cb4e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 23 00:21:49.839770 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:49.839738 2585 generic.go:356] "Generic (PLEG): container finished" podID="b4ae9351-6cb1-4305-8e63-bca4ee681df1" containerID="60700de09ec9a00fc86c2ae6141364c686ba2db8056cddbd7557156d9a9f7de8" exitCode=0 Jul 23 00:21:49.840122 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:49.839798 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" event={"ID":"b4ae9351-6cb1-4305-8e63-bca4ee681df1","Type":"ContainerDied","Data":"60700de09ec9a00fc86c2ae6141364c686ba2db8056cddbd7557156d9a9f7de8"} Jul 23 00:21:49.840199 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:49.840127 2585 scope.go:117] "RemoveContainer" containerID="60700de09ec9a00fc86c2ae6141364c686ba2db8056cddbd7557156d9a9f7de8" Jul 23 00:21:50.845593 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:50.845556 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mc4fh" event={"ID":"b4ae9351-6cb1-4305-8e63-bca4ee681df1","Type":"ContainerStarted","Data":"819f3bfc8eee1d73be44883fe674a62fe0175e9b8eb2ee1f700acf852551ae61"} Jul 23 00:21:55.865432 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:55.865396 2585 generic.go:356] "Generic (PLEG): container finished" podID="9161329a-2d0c-4b7b-bfdf-a82226410633" containerID="5b982d830bd47bd0928c37b596d7ea50353308dab17e26f05f4327699b70a164" exitCode=0 Jul 23 00:21:55.865860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:55.865488 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" event={"ID":"9161329a-2d0c-4b7b-bfdf-a82226410633","Type":"ContainerDied","Data":"5b982d830bd47bd0928c37b596d7ea50353308dab17e26f05f4327699b70a164"} Jul 23 00:21:55.865860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:55.865808 2585 scope.go:117] "RemoveContainer" containerID="5b982d830bd47bd0928c37b596d7ea50353308dab17e26f05f4327699b70a164" Jul 23 00:21:56.870978 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:56.870943 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2drrb" event={"ID":"9161329a-2d0c-4b7b-bfdf-a82226410633","Type":"ContainerStarted","Data":"3d944bf786b76e03ba2011402a29d6dc25fb494879d5038da2016b418b6d5172"} Jul 23 00:21:58.518283 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:21:58.518246 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" podUID="98d0690d-adb1-4eee-8d63-d9608764cb4e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 23 00:22:08.518269 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.518230 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" podUID="98d0690d-adb1-4eee-8d63-d9608764cb4e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 23 00:22:08.518731 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.518296 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" Jul 23 00:22:08.518854 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.518835 2585 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6c4ed24b415239c296fe26dd07012bdf521d29d4c1439645bc86fded68ee18fb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 23 00:22:08.518895 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.518876 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" podUID="98d0690d-adb1-4eee-8d63-d9608764cb4e" containerName="service-proxy" containerID="cri-o://6c4ed24b415239c296fe26dd07012bdf521d29d4c1439645bc86fded68ee18fb" gracePeriod=30 Jul 23 00:22:08.919235 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.919144 2585 generic.go:356] "Generic (PLEG): container finished" podID="98d0690d-adb1-4eee-8d63-d9608764cb4e" containerID="6c4ed24b415239c296fe26dd07012bdf521d29d4c1439645bc86fded68ee18fb" exitCode=2 Jul 23 00:22:08.919235 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.919194 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" event={"ID":"98d0690d-adb1-4eee-8d63-d9608764cb4e","Type":"ContainerDied","Data":"6c4ed24b415239c296fe26dd07012bdf521d29d4c1439645bc86fded68ee18fb"} Jul 23 00:22:08.919448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:08.919235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6697599ff6-v22mz" event={"ID":"98d0690d-adb1-4eee-8d63-d9608764cb4e","Type":"ContainerStarted","Data":"44c29fd65e02b00865394e26dcaf30548b4c2edccc1a2e11268b657e3dd76b6e"} Jul 23 00:22:20.918459 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:20.918418 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:20.974170 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:20.974136 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:25.273383 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:25.273298 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:25.360509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:25.360456 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:32.896138 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:32.896092 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:32.995689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:32.995657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:36.046854 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:36.046815 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:36.149786 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:36.149738 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:36.200759 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:36.200725 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:44.274008 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:44.273972 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:44.380134 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:44.380098 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:45.409417 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:45.409384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:45.485481 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:45.485395 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:59.474063 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:59.474010 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:59.527545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:22:59.527514 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:07.532235 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:07.532200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:07.600901 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:07.600873 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:38.863030 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:38.862989 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:39.671817 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:39.671780 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:42.455606 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:42.455572 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:42.556243 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:42.556195 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:43.545807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:43.545773 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:43.648669 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:43.648619 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:44.573051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:44.573018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:44.674860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:44.674829 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:46.791932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:46.791890 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:46.964308 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:46.964270 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:47.994509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:47.994444 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:49.050207 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:49.050177 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:49.137395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:49.137362 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:54.973664 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:54.973583 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:56.349781 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:56.349743 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:56.442941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:23:56.442899 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:02.947829 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:02.947797 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:03.045879 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:03.045847 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:03.293677 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:03.293620 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:05.112726 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:05.112693 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:10.832526 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:10.832489 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:10.933207 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:10.933166 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:21.134523 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:21.134481 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:21.223830 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:21.223792 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:21.397389 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:21.397301 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:21.525768 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:21.525733 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:21.716805 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:21.716716 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.055119 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.055087 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.135382 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135355 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m"] Jul 23 00:24:22.135741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135655 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4c4590-1220-4182-b2d7-d07077143423" containerName="registry" Jul 23 00:24:22.135741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135666 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4c4590-1220-4182-b2d7-d07077143423" containerName="registry" Jul 23 00:24:22.135741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135696 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75028301-5e65-4c43-8b06-28b6454f9820" containerName="registry" Jul 23 00:24:22.135741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135704 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="75028301-5e65-4c43-8b06-28b6454f9820" containerName="registry" Jul 23 00:24:22.135741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135714 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2574a9-6681-4810-814e-18238be3a296" containerName="console" Jul 23 00:24:22.135741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135719 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2574a9-6681-4810-814e-18238be3a296" containerName="console" Jul 23 00:24:22.135930 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135769 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae4c4590-1220-4182-b2d7-d07077143423" containerName="registry" Jul 23 00:24:22.135930 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135780 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2574a9-6681-4810-814e-18238be3a296" containerName="console" Jul 23 00:24:22.135930 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.135786 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="75028301-5e65-4c43-8b06-28b6454f9820" containerName="registry" Jul 23 00:24:22.137637 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.137621 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.141102 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.141082 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 00:24:22.141328 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.141309 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 00:24:22.141410 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.141317 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 00:24:22.141410 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.141341 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rqkdt\"" Jul 23 00:24:22.141410 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.141372 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 00:24:22.159718 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.159688 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m"] Jul 23 00:24:22.226865 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.226832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed1f769-bc32-473e-b7b0-245754cc103a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.227073 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.226875 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed1f769-bc32-473e-b7b0-245754cc103a-tmp\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.227073 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.226924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ed1f769-bc32-473e-b7b0-245754cc103a-manifests\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.227073 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.226977 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ws82\" (UniqueName: \"kubernetes.io/projected/0ed1f769-bc32-473e-b7b0-245754cc103a-kube-api-access-4ws82\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.227073 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.227007 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ed1f769-bc32-473e-b7b0-245754cc103a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.230800 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.230771 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.328251 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.328174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ws82\" (UniqueName: \"kubernetes.io/projected/0ed1f769-bc32-473e-b7b0-245754cc103a-kube-api-access-4ws82\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.328251 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.328213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ed1f769-bc32-473e-b7b0-245754cc103a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.328528 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.328489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed1f769-bc32-473e-b7b0-245754cc103a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.328607 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.328554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed1f769-bc32-473e-b7b0-245754cc103a-tmp\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.328607 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.328593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ed1f769-bc32-473e-b7b0-245754cc103a-manifests\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.328922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.328896 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ed1f769-bc32-473e-b7b0-245754cc103a-manifests\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.330622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.330592 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed1f769-bc32-473e-b7b0-245754cc103a-tmp\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.330826 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.330806 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ed1f769-bc32-473e-b7b0-245754cc103a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.330889 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.330809 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed1f769-bc32-473e-b7b0-245754cc103a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.340342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.340322 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ws82\" (UniqueName: \"kubernetes.io/projected/0ed1f769-bc32-473e-b7b0-245754cc103a-kube-api-access-4ws82\") pod \"opendatahub-operator-controller-manager-8844c5db8-fzf9m\" (UID: \"0ed1f769-bc32-473e-b7b0-245754cc103a\") " pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.358373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.358351 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.446810 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.446778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:22.456325 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.456155 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.556133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.556108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.577841 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.577819 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m"] Jul 23 00:24:22.579760 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:24:22.579696 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed1f769_bc32_473e_b7b0_245754cc103a.slice/crio-786f49118fb4e42a97268ce95c0273e3e058918e6691ef8f175baa9f561feea2 WatchSource:0}: Error finding container 786f49118fb4e42a97268ce95c0273e3e058918e6691ef8f175baa9f561feea2: Status 404 returned error can't find the container with id 786f49118fb4e42a97268ce95c0273e3e058918e6691ef8f175baa9f561feea2 Jul 23 00:24:22.619145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.619118 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.684236 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.684200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:22.785128 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:22.785095 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:23.453332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:23.453288 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" event={"ID":"0ed1f769-bc32-473e-b7b0-245754cc103a","Type":"ContainerStarted","Data":"786f49118fb4e42a97268ce95c0273e3e058918e6691ef8f175baa9f561feea2"} Jul 23 00:24:26.469327 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:26.469287 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" event={"ID":"0ed1f769-bc32-473e-b7b0-245754cc103a","Type":"ContainerStarted","Data":"0b931a9cd96325c61583bdec46d957148fa4f3a851ffb1b06f7351bc74c779ac"} Jul 23 00:24:27.474189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:27.474106 2585 generic.go:356] "Generic (PLEG): container finished" podID="0ed1f769-bc32-473e-b7b0-245754cc103a" containerID="0b931a9cd96325c61583bdec46d957148fa4f3a851ffb1b06f7351bc74c779ac" exitCode=0 Jul 23 00:24:27.474189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:27.474157 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" event={"ID":"0ed1f769-bc32-473e-b7b0-245754cc103a","Type":"ContainerDied","Data":"0b931a9cd96325c61583bdec46d957148fa4f3a851ffb1b06f7351bc74c779ac"} Jul 23 00:24:28.480758 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:28.480714 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" event={"ID":"0ed1f769-bc32-473e-b7b0-245754cc103a","Type":"ContainerStarted","Data":"65c8e4dc5d40378dc3e1d56ea390fffc17b37c52d53e8f049a3a6671538dae2d"} Jul 23 00:24:28.481237 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:28.480863 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:28.501882 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:28.501839 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" podStartSLOduration=1.675412753 podStartE2EDuration="6.501824993s" totalImagesPullingTime="3.752799758s" totalInitContainerRuntime="1.073612482s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:28.500556116 +0000 UTC m=+297.797251084" watchObservedRunningTime="2026-07-23 00:24:28.501824993 +0000 UTC m=+297.798519961" Jul 23 00:24:31.114036 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:31.113995 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:24:31.118697 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:31.118672 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:24:31.127256 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:31.127232 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 00:24:39.487105 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:39.487076 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8844c5db8-fzf9m" Jul 23 00:24:41.159892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.159833 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:41.502557 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.502528 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c7d578648-5x8gg"] Jul 23 00:24:41.504625 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.504609 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.506800 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.506771 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 00:24:41.506920 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.506775 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mqsp8\"" Jul 23 00:24:41.506920 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.506789 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 00:24:41.516830 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.516805 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c7d578648-5x8gg"] Jul 23 00:24:41.597285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.597252 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpmsb\" (UniqueName: \"kubernetes.io/projected/9d05b2c8-049b-427a-ad50-68cdc2e6068b-kube-api-access-mpmsb\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.597432 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.597289 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d05b2c8-049b-427a-ad50-68cdc2e6068b-tmp\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.597432 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.597317 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d05b2c8-049b-427a-ad50-68cdc2e6068b-tls-certs\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.698441 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.698414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d05b2c8-049b-427a-ad50-68cdc2e6068b-tmp\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.698624 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.698485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d05b2c8-049b-427a-ad50-68cdc2e6068b-tls-certs\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.698696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.698645 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpmsb\" (UniqueName: \"kubernetes.io/projected/9d05b2c8-049b-427a-ad50-68cdc2e6068b-kube-api-access-mpmsb\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.700684 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.700663 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d05b2c8-049b-427a-ad50-68cdc2e6068b-tmp\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.700918 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.700898 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d05b2c8-049b-427a-ad50-68cdc2e6068b-tls-certs\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.713365 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.713338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpmsb\" (UniqueName: \"kubernetes.io/projected/9d05b2c8-049b-427a-ad50-68cdc2e6068b-kube-api-access-mpmsb\") pod \"kube-auth-proxy-c7d578648-5x8gg\" (UID: \"9d05b2c8-049b-427a-ad50-68cdc2e6068b\") " pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.814588 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.814499 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" Jul 23 00:24:41.895165 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.895132 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g8mtv"] Jul 23 00:24:41.898095 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.898072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:41.900084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.900059 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qjrxc\"" Jul 23 00:24:41.900558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.900538 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 00:24:41.909648 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.909557 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g8mtv"] Jul 23 00:24:41.948780 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.948757 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c7d578648-5x8gg"] Jul 23 00:24:41.950965 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:24:41.950940 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d05b2c8_049b_427a_ad50_68cdc2e6068b.slice/crio-2c9312ac26531c4ab74f2f7bb20c2fdab9a709775463e55030f5e654cb6aca15 WatchSource:0}: Error finding container 2c9312ac26531c4ab74f2f7bb20c2fdab9a709775463e55030f5e654cb6aca15: Status 404 returned error can't find the container with id 2c9312ac26531c4ab74f2f7bb20c2fdab9a709775463e55030f5e654cb6aca15 Jul 23 00:24:41.953108 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:41.953087 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:24:42.000759 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.000733 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7067c92-8c8e-4880-93f8-2d88cecd4009-cert\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.000879 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.000789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp5h\" (UniqueName: \"kubernetes.io/projected/d7067c92-8c8e-4880-93f8-2d88cecd4009-kube-api-access-sfp5h\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.101897 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.101828 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7067c92-8c8e-4880-93f8-2d88cecd4009-cert\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.101897 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.101873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp5h\" (UniqueName: \"kubernetes.io/projected/d7067c92-8c8e-4880-93f8-2d88cecd4009-kube-api-access-sfp5h\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.102124 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:42.101983 2585 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 00:24:42.102124 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:42.102045 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7067c92-8c8e-4880-93f8-2d88cecd4009-cert podName:d7067c92-8c8e-4880-93f8-2d88cecd4009 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:42.602026553 +0000 UTC m=+311.898721513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d7067c92-8c8e-4880-93f8-2d88cecd4009-cert") pod "odh-model-controller-7998c67dc4-g8mtv" (UID: "d7067c92-8c8e-4880-93f8-2d88cecd4009") : secret "odh-model-controller-webhook-cert" not found Jul 23 00:24:42.113068 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.113042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp5h\" (UniqueName: \"kubernetes.io/projected/d7067c92-8c8e-4880-93f8-2d88cecd4009-kube-api-access-sfp5h\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.486583 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.486548 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:42.540492 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.540441 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" event={"ID":"9d05b2c8-049b-427a-ad50-68cdc2e6068b","Type":"ContainerStarted","Data":"2c9312ac26531c4ab74f2f7bb20c2fdab9a709775463e55030f5e654cb6aca15"} Jul 23 00:24:42.605825 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.605793 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7067c92-8c8e-4880-93f8-2d88cecd4009-cert\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.608437 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.608416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7067c92-8c8e-4880-93f8-2d88cecd4009-cert\") pod \"odh-model-controller-7998c67dc4-g8mtv\" (UID: \"d7067c92-8c8e-4880-93f8-2d88cecd4009\") " pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.723637 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.723608 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jqk76"] Jul 23 00:24:42.725950 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.725935 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:42.728266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.728244 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 00:24:42.740664 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.740614 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jqk76"] Jul 23 00:24:42.749960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.749933 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:42.808109 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.808077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:42.808267 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.808116 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr68x\" (UniqueName: \"kubernetes.io/projected/5bf6f6ed-da14-4dbe-a62f-c2207c069042-kube-api-access-cr68x\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:42.810986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.810963 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:42.909450 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.909417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:42.909626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.909498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr68x\" (UniqueName: \"kubernetes.io/projected/5bf6f6ed-da14-4dbe-a62f-c2207c069042-kube-api-access-cr68x\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:42.909626 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:42.909575 2585 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 23 00:24:42.909742 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:42.909652 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs podName:5bf6f6ed-da14-4dbe-a62f-c2207c069042 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:43.409629151 +0000 UTC m=+312.706324112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs") pod "model-serving-api-54d89bd5fc-jqk76" (UID: "5bf6f6ed-da14-4dbe-a62f-c2207c069042") : secret "model-serving-api-tls" not found Jul 23 00:24:42.920847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.920789 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr68x\" (UniqueName: \"kubernetes.io/projected/5bf6f6ed-da14-4dbe-a62f-c2207c069042-kube-api-access-cr68x\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:42.928540 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.928426 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:42.935274 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.935059 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6"] Jul 23 00:24:42.938758 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.938736 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:42.941594 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.941307 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 00:24:42.941594 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.941518 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 00:24:42.941818 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.941695 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 00:24:42.941903 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.941854 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hxtfh\"" Jul 23 00:24:42.942051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.942011 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 00:24:42.951777 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.951713 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6"] Jul 23 00:24:42.971858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:42.971812 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g8mtv"] Jul 23 00:24:42.976756 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:24:42.976727 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7067c92_8c8e_4880_93f8_2d88cecd4009.slice/crio-50c60ecb50ab0201d3767760f4ef45c904d0eba631a2d3c1d8796f16968c92cc WatchSource:0}: Error finding container 50c60ecb50ab0201d3767760f4ef45c904d0eba631a2d3c1d8796f16968c92cc: Status 404 returned error can't find the container with id 50c60ecb50ab0201d3767760f4ef45c904d0eba631a2d3c1d8796f16968c92cc Jul 23 00:24:43.111558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111472 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.111558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111540 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-cacerts\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.111756 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111578 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9a192a-344b-4518-ab60-dc08b7ba7b03-local-certs\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.111756 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-token\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.111756 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111707 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glc67\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-kube-api-access-glc67\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.111901 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111779 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.111901 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.111814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.181298 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.181259 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:43.212271 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212237 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-cacerts\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.212453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9a192a-344b-4518-ab60-dc08b7ba7b03-local-certs\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.212453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212357 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-token\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.212453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212383 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glc67\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-kube-api-access-glc67\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.212453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212430 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.212693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.212693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.212519 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.213726 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.213696 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.215558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.215532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-cacerts\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.215655 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.215569 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.215696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.215644 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.215859 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.215841 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9a192a-344b-4518-ab60-dc08b7ba7b03-local-certs\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.220632 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.220613 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-token\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.220738 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.220633 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glc67\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-kube-api-access-glc67\") pod \"istiod-openshift-gateway-7574459656-sbwg6\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.253665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.253642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:43.381603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.381573 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6"] Jul 23 00:24:43.383632 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:24:43.383604 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9a192a_344b_4518_ab60_dc08b7ba7b03.slice/crio-2561bb8a2af9f89a4fe011a28156454c553205a91148f07b0048b48459d2a8fd WatchSource:0}: Error finding container 2561bb8a2af9f89a4fe011a28156454c553205a91148f07b0048b48459d2a8fd: Status 404 returned error can't find the container with id 2561bb8a2af9f89a4fe011a28156454c553205a91148f07b0048b48459d2a8fd Jul 23 00:24:43.414547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.414521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:43.414712 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:43.414693 2585 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 23 00:24:43.414806 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:43.414794 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs podName:5bf6f6ed-da14-4dbe-a62f-c2207c069042 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:44.414772345 +0000 UTC m=+313.711467293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs") pod "model-serving-api-54d89bd5fc-jqk76" (UID: "5bf6f6ed-da14-4dbe-a62f-c2207c069042") : secret "model-serving-api-tls" not found Jul 23 00:24:43.545419 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.545378 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:43.546055 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.546022 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" event={"ID":"d7067c92-8c8e-4880-93f8-2d88cecd4009","Type":"ContainerStarted","Data":"50c60ecb50ab0201d3767760f4ef45c904d0eba631a2d3c1d8796f16968c92cc"} Jul 23 00:24:43.547155 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.547137 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" event={"ID":"1c9a192a-344b-4518-ab60-dc08b7ba7b03","Type":"ContainerStarted","Data":"2561bb8a2af9f89a4fe011a28156454c553205a91148f07b0048b48459d2a8fd"} Jul 23 00:24:43.844590 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:43.844540 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:44.423113 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:44.423057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:44.426429 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:44.426402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf6f6ed-da14-4dbe-a62f-c2207c069042-tls-certs\") pod \"model-serving-api-54d89bd5fc-jqk76\" (UID: \"5bf6f6ed-da14-4dbe-a62f-c2207c069042\") " pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:44.536296 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:44.536118 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:47.501444 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:47.501401 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:24:47.501880 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:47.501505 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:24:47.664691 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:47.664664 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jqk76"] Jul 23 00:24:47.667110 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:24:47.667064 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf6f6ed_da14_4dbe_a62f_c2207c069042.slice/crio-72be38c41e24c59e6bf65702e310a5cf73620b6fa26d752647ba4c915037a06d WatchSource:0}: Error finding container 72be38c41e24c59e6bf65702e310a5cf73620b6fa26d752647ba4c915037a06d: Status 404 returned error can't find the container with id 72be38c41e24c59e6bf65702e310a5cf73620b6fa26d752647ba4c915037a06d Jul 23 00:24:48.450219 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.450185 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:48.552784 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.552742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:48.579122 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.579090 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" event={"ID":"d7067c92-8c8e-4880-93f8-2d88cecd4009","Type":"ContainerStarted","Data":"790ecf23378a8936eeccf8d48c5ef7d9a38fd01ff4806489e94402824bf21ae6"} Jul 23 00:24:48.579300 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.579199 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:24:48.580292 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.580262 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" event={"ID":"5bf6f6ed-da14-4dbe-a62f-c2207c069042","Type":"ContainerStarted","Data":"72be38c41e24c59e6bf65702e310a5cf73620b6fa26d752647ba4c915037a06d"} Jul 23 00:24:48.581936 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.581907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" event={"ID":"1c9a192a-344b-4518-ab60-dc08b7ba7b03","Type":"ContainerStarted","Data":"43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51"} Jul 23 00:24:48.582050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.582033 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:48.583707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.583683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" event={"ID":"9d05b2c8-049b-427a-ad50-68cdc2e6068b","Type":"ContainerStarted","Data":"cc4cd1917e706f46c07f6ee357ccc1a61e2e84dd5af186fd16fb4fe3fc449d25"} Jul 23 00:24:48.584100 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.584066 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-sbwg6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:24:48.584207 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.584112 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:24:48.599508 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.599444 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" podStartSLOduration=3.067898414 podStartE2EDuration="7.599427202s" totalImagesPullingTime="4.531528788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:48.598689349 +0000 UTC m=+317.895384342" watchObservedRunningTime="2026-07-23 00:24:48.599427202 +0000 UTC m=+317.896122171" Jul 23 00:24:48.616516 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.616449 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c7d578648-5x8gg" podStartSLOduration=2.081757776 podStartE2EDuration="7.616430966s" totalImagesPullingTime="5.53467319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:48.614756474 +0000 UTC m=+317.911451445" watchObservedRunningTime="2026-07-23 00:24:48.616430966 +0000 UTC m=+317.913125936" Jul 23 00:24:48.644271 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.644010 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" podStartSLOduration=2.528845838 podStartE2EDuration="6.64398664s" totalImagesPullingTime="4.115140802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:48.643137729 +0000 UTC m=+317.939832700" watchObservedRunningTime="2026-07-23 00:24:48.64398664 +0000 UTC m=+317.940681609" Jul 23 00:24:48.719059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.718985 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh"] Jul 23 00:24:48.723120 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.723085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.725927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.725899 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lm8zm\"" Jul 23 00:24:48.738688 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.738664 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh"] Jul 23 00:24:48.762568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.762530 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:48.867319 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867330 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4853630f-e743-4b5d-a3a1-919e3280e496-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867360 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4853630f-e743-4b5d-a3a1-919e3280e496-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867396 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4853630f-e743-4b5d-a3a1-919e3280e496-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867425 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867561 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867587 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.867681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.867643 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkdw\" (UniqueName: \"kubernetes.io/projected/4853630f-e743-4b5d-a3a1-919e3280e496-kube-api-access-5dkdw\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969008 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.968971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969221 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4853630f-e743-4b5d-a3a1-919e3280e496-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969284 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4853630f-e743-4b5d-a3a1-919e3280e496-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969325 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4853630f-e743-4b5d-a3a1-919e3280e496-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969328 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969420 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969501 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969452 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969491 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkdw\" (UniqueName: \"kubernetes.io/projected/4853630f-e743-4b5d-a3a1-919e3280e496-kube-api-access-5dkdw\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969833 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969752 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.969980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.969959 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.970051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.970029 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.970254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.970231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4853630f-e743-4b5d-a3a1-919e3280e496-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.972303 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.972281 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4853630f-e743-4b5d-a3a1-919e3280e496-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.972408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.972338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4853630f-e743-4b5d-a3a1-919e3280e496-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.990767 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.990742 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4853630f-e743-4b5d-a3a1-919e3280e496-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:48.991055 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:48.991026 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkdw\" (UniqueName: \"kubernetes.io/projected/4853630f-e743-4b5d-a3a1-919e3280e496-kube-api-access-5dkdw\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hcwkh\" (UID: \"4853630f-e743-4b5d-a3a1-919e3280e496\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:49.037394 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.037359 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:49.047060 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.047018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:49.150838 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.150765 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:49.527533 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.527508 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh"] Jul 23 00:24:49.529713 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:24:49.529666 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4853630f_e743_4b5d_a3a1_919e3280e496.slice/crio-e1d229696e1f89add6e5d6ec6d8dacf03221f3dc35ae75b42aec3b0d21507535 WatchSource:0}: Error finding container e1d229696e1f89add6e5d6ec6d8dacf03221f3dc35ae75b42aec3b0d21507535: Status 404 returned error can't find the container with id e1d229696e1f89add6e5d6ec6d8dacf03221f3dc35ae75b42aec3b0d21507535 Jul 23 00:24:49.589452 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.589419 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" event={"ID":"5bf6f6ed-da14-4dbe-a62f-c2207c069042","Type":"ContainerStarted","Data":"21256ffb63606640c3e9bdaeede0864dd29b5c4dd84eac4d269f1c9c3921e69d"} Jul 23 00:24:49.589870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.589611 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:24:49.590812 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.590784 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" event={"ID":"4853630f-e743-4b5d-a3a1-919e3280e496","Type":"ContainerStarted","Data":"e1d229696e1f89add6e5d6ec6d8dacf03221f3dc35ae75b42aec3b0d21507535"} Jul 23 00:24:49.591660 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.591641 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:49.606874 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:49.606834 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" podStartSLOduration=5.829646576 podStartE2EDuration="7.606821242s" totalImagesPullingTime="1.777174666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:49.604539866 +0000 UTC m=+318.901234834" watchObservedRunningTime="2026-07-23 00:24:49.606821242 +0000 UTC m=+318.903516210" Jul 23 00:24:51.173977 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.173912 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:51.232366 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.231394 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj"] Jul 23 00:24:51.239133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.239063 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:51.247615 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.245108 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bhgm5\"" Jul 23 00:24:51.247615 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.246930 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 00:24:51.248241 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.248153 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj"] Jul 23 00:24:51.278660 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.278592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:51.393184 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.393147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfhb\" (UniqueName: \"kubernetes.io/projected/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-kube-api-access-jqfhb\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:51.393367 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.393196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:51.478490 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.478377 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:51.494531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.494490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:51.494700 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:51.494647 2585 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 23 00:24:51.494756 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:51.494716 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert podName:d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:51.994697304 +0000 UTC m=+321.291392253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert") pod "llmisvc-controller-manager-777c6bd7d5-kzldj" (UID: "d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9") : secret "llmisvc-webhook-server-cert" not found Jul 23 00:24:51.494943 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.494646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfhb\" (UniqueName: \"kubernetes.io/projected/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-kube-api-access-jqfhb\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:51.505152 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.505071 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfhb\" (UniqueName: \"kubernetes.io/projected/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-kube-api-access-jqfhb\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:51.601492 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.601303 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:51.999806 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:51.999770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:52.000010 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:51.999963 2585 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 23 00:24:52.000078 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:52.000035 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert podName:d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:53.000013576 +0000 UTC m=+322.296708546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert") pod "llmisvc-controller-manager-777c6bd7d5-kzldj" (UID: "d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9") : secret "llmisvc-webhook-server-cert" not found Jul 23 00:24:52.053103 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.053057 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:52.236822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.236793 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:52.296291 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.296262 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:24:52.296378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.296330 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:24:52.296378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.296365 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:24:52.614061 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.613972 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" event={"ID":"4853630f-e743-4b5d-a3a1-919e3280e496","Type":"ContainerStarted","Data":"e68209b9b8c634d4460916ec59ec418401c283f4689c4798a821b585ff0bd144"} Jul 23 00:24:52.634680 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:52.634626 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" podStartSLOduration=1.8714927879999999 podStartE2EDuration="4.634609028s" totalImagesPullingTime="2.76311624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:52.632301712 +0000 UTC m=+321.928996679" watchObservedRunningTime="2026-07-23 00:24:52.634609028 +0000 UTC m=+321.931303996" Jul 23 00:24:53.009660 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.009624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:53.012036 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.012005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzldj\" (UID: \"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:53.037864 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.037839 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:53.039253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.039233 2585 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 23 00:24:53.039307 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.039278 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" podUID="4853630f-e743-4b5d-a3a1-919e3280e496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 23 00:24:53.057460 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.057435 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:53.198920 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.198888 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj"] Jul 23 00:24:53.619642 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:53.619607 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" event={"ID":"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9","Type":"ContainerStarted","Data":"df211945da3c3a698bfbd6c6c9fbfdafb954f8924e979fa18877f853acf5410b"} Jul 23 00:24:54.043021 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:54.042993 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:54.624966 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:54.624933 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:54.626035 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:54.626008 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hcwkh" Jul 23 00:24:55.543307 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.543274 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6"] Jul 23 00:24:55.543518 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.543492 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerName="discovery" containerID="cri-o://43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51" gracePeriod=30 Jul 23 00:24:55.591340 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.591310 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-sbwg6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:8080/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" start-of-body= Jul 23 00:24:55.591485 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.591377 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.33:8080/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Jul 23 00:24:55.805829 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.805771 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:55.833034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.832994 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-dns-cert\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.833215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.833175 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-kubeconfig\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.833289 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.833256 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glc67\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-kube-api-access-glc67\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.833355 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.833296 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-cacerts\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.833415 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.833351 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-ca-configmap\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.833415 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.833400 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9a192a-344b-4518-ab60-dc08b7ba7b03-local-certs\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.833575 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.833423 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-token\") pod \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\" (UID: \"1c9a192a-344b-4518-ab60-dc08b7ba7b03\") " Jul 23 00:24:55.834046 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.834014 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:24:55.836216 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.836177 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:24:55.836432 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.836403 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:24:55.836432 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.836416 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-cacerts" (OuterVolumeSpecName: "cacerts") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:24:55.836829 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.836806 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:24:55.837306 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.837279 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9a192a-344b-4518-ab60-dc08b7ba7b03-local-certs" (OuterVolumeSpecName: "local-certs") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:24:55.837408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.837367 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-kube-api-access-glc67" (OuterVolumeSpecName: "kube-api-access-glc67") pod "1c9a192a-344b-4518-ab60-dc08b7ba7b03" (UID: "1c9a192a-344b-4518-ab60-dc08b7ba7b03"). InnerVolumeSpecName "kube-api-access-glc67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:24:55.934332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934303 2585 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9a192a-344b-4518-ab60-dc08b7ba7b03-local-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:55.934332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934330 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:55.934531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934343 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-dns-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:55.934531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934356 2585 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-kubeconfig\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:55.934531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934369 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glc67\" (UniqueName: \"kubernetes.io/projected/1c9a192a-344b-4518-ab60-dc08b7ba7b03-kube-api-access-glc67\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:55.934531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934380 2585 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c9a192a-344b-4518-ab60-dc08b7ba7b03-cacerts\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:55.934531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:55.934391 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c9a192a-344b-4518-ab60-dc08b7ba7b03-istio-csr-ca-configmap\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:24:56.635159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.635125 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" event={"ID":"d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9","Type":"ContainerStarted","Data":"768792e1747590487657fa4876eeea66a5cdadcfd223e732f9e1cf27a8f45b40"} Jul 23 00:24:56.635286 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.635194 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:24:56.636410 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.636385 2585 generic.go:356] "Generic (PLEG): container finished" podID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerID="43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51" exitCode=0 Jul 23 00:24:56.636497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.636417 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" Jul 23 00:24:56.636497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.636448 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" event={"ID":"1c9a192a-344b-4518-ab60-dc08b7ba7b03","Type":"ContainerDied","Data":"43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51"} Jul 23 00:24:56.636580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.636500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6" event={"ID":"1c9a192a-344b-4518-ab60-dc08b7ba7b03","Type":"ContainerDied","Data":"2561bb8a2af9f89a4fe011a28156454c553205a91148f07b0048b48459d2a8fd"} Jul 23 00:24:56.636580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.636522 2585 scope.go:117] "RemoveContainer" containerID="43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51" Jul 23 00:24:56.645163 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.645145 2585 scope.go:117] "RemoveContainer" containerID="43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51" Jul 23 00:24:56.645407 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:24:56.645389 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51\": container with ID starting with 43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51 not found: ID does not exist" containerID="43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51" Jul 23 00:24:56.645455 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.645414 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51"} err="failed to get container status \"43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51\": rpc error: code = NotFound desc = could not find container \"43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51\": container with ID starting with 43bfac932cc76f792ac7b95596fa7e9519b4420c3440e26f15c4d9c74a6d9e51 not found: ID does not exist" Jul 23 00:24:56.650104 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.650070 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" podStartSLOduration=2.363267299 podStartE2EDuration="5.650057087s" totalImagesPullingTime="3.286789788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:56.649872629 +0000 UTC m=+325.946567596" watchObservedRunningTime="2026-07-23 00:24:56.650057087 +0000 UTC m=+325.946752056" Jul 23 00:24:56.664442 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.664417 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6"] Jul 23 00:24:56.670962 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:56.670940 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sbwg6"] Jul 23 00:24:57.412018 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:57.411986 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" path="/var/lib/kubelet/pods/1c9a192a-344b-4518-ab60-dc08b7ba7b03/volumes" Jul 23 00:24:59.593217 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:24:59.593188 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-g8mtv" Jul 23 00:25:00.601016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:00.600989 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jqk76" Jul 23 00:25:27.650145 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:27.650074 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzldj" Jul 23 00:25:42.684551 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.684515 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:25:42.761434 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.761402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:25:42.809020 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.808985 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjklv"] Jul 23 00:25:42.809405 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.809386 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerName="discovery" Jul 23 00:25:42.809507 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.809406 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerName="discovery" Jul 23 00:25:42.809569 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.809526 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9a192a-344b-4518-ab60-dc08b7ba7b03" containerName="discovery" Jul 23 00:25:42.813791 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.813770 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:42.816448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.816417 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 00:25:42.816448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.816439 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-lbncf\"" Jul 23 00:25:42.816649 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.816487 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 00:25:42.820921 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.820893 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjklv"] Jul 23 00:25:42.853182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.853147 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:25:42.927235 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:42.927203 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjts\" (UniqueName: \"kubernetes.io/projected/92c32595-a69d-4a4a-bf0b-f3f449fedb31-kube-api-access-sbjts\") pod \"kuadrant-operator-catalog-wjklv\" (UID: \"92c32595-a69d-4a4a-bf0b-f3f449fedb31\") " pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:43.028701 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.028669 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjts\" (UniqueName: \"kubernetes.io/projected/92c32595-a69d-4a4a-bf0b-f3f449fedb31-kube-api-access-sbjts\") pod \"kuadrant-operator-catalog-wjklv\" (UID: \"92c32595-a69d-4a4a-bf0b-f3f449fedb31\") " pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:43.036402 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.036378 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjts\" (UniqueName: \"kubernetes.io/projected/92c32595-a69d-4a4a-bf0b-f3f449fedb31-kube-api-access-sbjts\") pod \"kuadrant-operator-catalog-wjklv\" (UID: \"92c32595-a69d-4a4a-bf0b-f3f449fedb31\") " pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:43.125989 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.125955 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:43.178214 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.178180 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjklv"] Jul 23 00:25:43.253038 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.253013 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjklv"] Jul 23 00:25:43.254855 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:25:43.254822 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c32595_a69d_4a4a_bf0b_f3f449fedb31.slice/crio-cdb53b3252d77aaa577cef42869bf47365dea9cf0648890d8c8317b1dcb8e30e WatchSource:0}: Error finding container cdb53b3252d77aaa577cef42869bf47365dea9cf0648890d8c8317b1dcb8e30e: Status 404 returned error can't find the container with id cdb53b3252d77aaa577cef42869bf47365dea9cf0648890d8c8317b1dcb8e30e Jul 23 00:25:43.384611 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.384538 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2dh4f"] Jul 23 00:25:43.387538 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.387522 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:43.395044 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.395022 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2dh4f"] Jul 23 00:25:43.532225 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.532196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9fs\" (UniqueName: \"kubernetes.io/projected/cfe51fe7-3af6-4467-b549-5c3a8f478d01-kube-api-access-6p9fs\") pod \"kuadrant-operator-catalog-2dh4f\" (UID: \"cfe51fe7-3af6-4467-b549-5c3a8f478d01\") " pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:43.633116 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.633083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9fs\" (UniqueName: \"kubernetes.io/projected/cfe51fe7-3af6-4467-b549-5c3a8f478d01-kube-api-access-6p9fs\") pod \"kuadrant-operator-catalog-2dh4f\" (UID: \"cfe51fe7-3af6-4467-b549-5c3a8f478d01\") " pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:43.640637 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.640582 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9fs\" (UniqueName: \"kubernetes.io/projected/cfe51fe7-3af6-4467-b549-5c3a8f478d01-kube-api-access-6p9fs\") pod \"kuadrant-operator-catalog-2dh4f\" (UID: \"cfe51fe7-3af6-4467-b549-5c3a8f478d01\") " pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:43.697576 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.697542 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:43.855008 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.854970 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" event={"ID":"92c32595-a69d-4a4a-bf0b-f3f449fedb31","Type":"ContainerStarted","Data":"cdb53b3252d77aaa577cef42869bf47365dea9cf0648890d8c8317b1dcb8e30e"} Jul 23 00:25:43.857916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:43.857862 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2dh4f"] Jul 23 00:25:43.888526 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:25:43.888490 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe51fe7_3af6_4467_b549_5c3a8f478d01.slice/crio-78f7d8b839297eb38379d59e8ba71f66bcb13214879ac8b9eb3b88634c05ee13 WatchSource:0}: Error finding container 78f7d8b839297eb38379d59e8ba71f66bcb13214879ac8b9eb3b88634c05ee13: Status 404 returned error can't find the container with id 78f7d8b839297eb38379d59e8ba71f66bcb13214879ac8b9eb3b88634c05ee13 Jul 23 00:25:44.863231 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:44.863186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" event={"ID":"cfe51fe7-3af6-4467-b549-5c3a8f478d01","Type":"ContainerStarted","Data":"78f7d8b839297eb38379d59e8ba71f66bcb13214879ac8b9eb3b88634c05ee13"} Jul 23 00:25:45.869815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:45.869771 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" event={"ID":"92c32595-a69d-4a4a-bf0b-f3f449fedb31","Type":"ContainerStarted","Data":"1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2"} Jul 23 00:25:45.869815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:45.869796 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" podUID="92c32595-a69d-4a4a-bf0b-f3f449fedb31" containerName="registry-server" containerID="cri-o://1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2" gracePeriod=2 Jul 23 00:25:45.871201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:45.871179 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" event={"ID":"cfe51fe7-3af6-4467-b549-5c3a8f478d01","Type":"ContainerStarted","Data":"c48cade8d87e97a18728b42e06d6383e52e42fa05e758e527c4b5dd304cfbbeb"} Jul 23 00:25:45.884870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:45.884827 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" podStartSLOduration=1.874525186 podStartE2EDuration="3.884812748s" totalImagesPullingTime="2.010287562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:25:45.883328038 +0000 UTC m=+375.180023009" watchObservedRunningTime="2026-07-23 00:25:45.884812748 +0000 UTC m=+375.181507726" Jul 23 00:25:45.898061 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:45.898013 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" podStartSLOduration=1.525559377 podStartE2EDuration="2.897995813s" totalImagesPullingTime="1.372436436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:25:45.897525977 +0000 UTC m=+375.194220955" watchObservedRunningTime="2026-07-23 00:25:45.897995813 +0000 UTC m=+375.194690784" Jul 23 00:25:46.116611 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.116589 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:46.156119 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.156049 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbjts\" (UniqueName: \"kubernetes.io/projected/92c32595-a69d-4a4a-bf0b-f3f449fedb31-kube-api-access-sbjts\") pod \"92c32595-a69d-4a4a-bf0b-f3f449fedb31\" (UID: \"92c32595-a69d-4a4a-bf0b-f3f449fedb31\") " Jul 23 00:25:46.158154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.158134 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c32595-a69d-4a4a-bf0b-f3f449fedb31-kube-api-access-sbjts" (OuterVolumeSpecName: "kube-api-access-sbjts") pod "92c32595-a69d-4a4a-bf0b-f3f449fedb31" (UID: "92c32595-a69d-4a4a-bf0b-f3f449fedb31"). InnerVolumeSpecName "kube-api-access-sbjts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:25:46.256672 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.256639 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbjts\" (UniqueName: \"kubernetes.io/projected/92c32595-a69d-4a4a-bf0b-f3f449fedb31-kube-api-access-sbjts\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:25:46.877252 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.877216 2585 generic.go:356] "Generic (PLEG): container finished" podID="92c32595-a69d-4a4a-bf0b-f3f449fedb31" containerID="1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2" exitCode=0 Jul 23 00:25:46.877685 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.877263 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" Jul 23 00:25:46.877685 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.877302 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" event={"ID":"92c32595-a69d-4a4a-bf0b-f3f449fedb31","Type":"ContainerDied","Data":"1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2"} Jul 23 00:25:46.877685 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.877357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wjklv" event={"ID":"92c32595-a69d-4a4a-bf0b-f3f449fedb31","Type":"ContainerDied","Data":"cdb53b3252d77aaa577cef42869bf47365dea9cf0648890d8c8317b1dcb8e30e"} Jul 23 00:25:46.877685 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.877378 2585 scope.go:117] "RemoveContainer" containerID="1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2" Jul 23 00:25:46.886935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.886914 2585 scope.go:117] "RemoveContainer" containerID="1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2" Jul 23 00:25:46.887189 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:25:46.887167 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2\": container with ID starting with 1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2 not found: ID does not exist" containerID="1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2" Jul 23 00:25:46.887292 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.887198 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2"} err="failed to get container status \"1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2\": rpc error: code = NotFound desc = could not find container \"1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2\": container with ID starting with 1ff60afb63de9ddf98fbedaa64364e065c464107e986edc5a7679b35bdf40ca2 not found: ID does not exist" Jul 23 00:25:46.898388 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.898362 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjklv"] Jul 23 00:25:46.906245 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:46.906224 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjklv"] Jul 23 00:25:47.411067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:47.411034 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c32595-a69d-4a4a-bf0b-f3f449fedb31" path="/var/lib/kubelet/pods/92c32595-a69d-4a4a-bf0b-f3f449fedb31/volumes" Jul 23 00:25:53.698054 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:53.698015 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:53.698544 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:53.698080 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:53.721094 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:53.721061 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:53.932139 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:53.932111 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2dh4f" Jul 23 00:25:56.581444 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:25:56.581410 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:11.889311 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:11.889269 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:14.513886 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.513851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc"] Jul 23 00:26:14.514358 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.514190 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c32595-a69d-4a4a-bf0b-f3f449fedb31" containerName="registry-server" Jul 23 00:26:14.514358 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.514201 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c32595-a69d-4a4a-bf0b-f3f449fedb31" containerName="registry-server" Jul 23 00:26:14.514358 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.514260 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c32595-a69d-4a4a-bf0b-f3f449fedb31" containerName="registry-server" Jul 23 00:26:14.520019 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.519999 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:14.523127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.523109 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lqrff\"" Jul 23 00:26:14.523127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.523119 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 23 00:26:14.533776 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.533751 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc"] Jul 23 00:26:14.594046 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.594020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zks67\" (UniqueName: \"kubernetes.io/projected/acd91885-e0e5-4c18-9506-1e6d6d967dba-kube-api-access-zks67\") pod \"dns-operator-controller-manager-864dd78c4c-pcndc\" (UID: \"acd91885-e0e5-4c18-9506-1e6d6d967dba\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:14.695414 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.695383 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zks67\" (UniqueName: \"kubernetes.io/projected/acd91885-e0e5-4c18-9506-1e6d6d967dba-kube-api-access-zks67\") pod \"dns-operator-controller-manager-864dd78c4c-pcndc\" (UID: \"acd91885-e0e5-4c18-9506-1e6d6d967dba\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:14.706964 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.706937 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zks67\" (UniqueName: \"kubernetes.io/projected/acd91885-e0e5-4c18-9506-1e6d6d967dba-kube-api-access-zks67\") pod \"dns-operator-controller-manager-864dd78c4c-pcndc\" (UID: \"acd91885-e0e5-4c18-9506-1e6d6d967dba\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:14.829854 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.829782 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:14.983045 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:14.983014 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc"] Jul 23 00:26:14.986186 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:14.986149 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd91885_e0e5_4c18_9506_1e6d6d967dba.slice/crio-b9a774923aa350a2f3ec7199da27af522734da0ff47009c2a2568a2332ceb1df WatchSource:0}: Error finding container b9a774923aa350a2f3ec7199da27af522734da0ff47009c2a2568a2332ceb1df: Status 404 returned error can't find the container with id b9a774923aa350a2f3ec7199da27af522734da0ff47009c2a2568a2332ceb1df Jul 23 00:26:15.011510 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:15.011458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" event={"ID":"acd91885-e0e5-4c18-9506-1e6d6d967dba","Type":"ContainerStarted","Data":"b9a774923aa350a2f3ec7199da27af522734da0ff47009c2a2568a2332ceb1df"} Jul 23 00:26:15.584407 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:15.584375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:18.026377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:18.026344 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" event={"ID":"acd91885-e0e5-4c18-9506-1e6d6d967dba","Type":"ContainerStarted","Data":"c88022edecbaa91269d0649693576804d9a296df870744806be06e901ea3b519"} Jul 23 00:26:18.026793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:18.026411 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:18.051050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:18.050943 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" podStartSLOduration=2.055723965 podStartE2EDuration="4.050925487s" totalImagesPullingTime="1.995201522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:18.049264108 +0000 UTC m=+407.345959086" watchObservedRunningTime="2026-07-23 00:26:18.050925487 +0000 UTC m=+407.347620457" Jul 23 00:26:18.529791 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:18.529761 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:19.957740 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:19.957689 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:20.267650 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.267614 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z"] Jul 23 00:26:20.271060 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.271043 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.273245 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.273225 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2tgzd\"" Jul 23 00:26:20.285747 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.285722 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z"] Jul 23 00:26:20.348809 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.348770 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnk5\" (UniqueName: \"kubernetes.io/projected/4e2e87b0-476b-40e9-873b-cae6a3e51cac-kube-api-access-8cnk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.348937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.348834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e2e87b0-476b-40e9-873b-cae6a3e51cac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.450172 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.450135 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnk5\" (UniqueName: \"kubernetes.io/projected/4e2e87b0-476b-40e9-873b-cae6a3e51cac-kube-api-access-8cnk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.450323 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.450181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e2e87b0-476b-40e9-873b-cae6a3e51cac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.450521 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.450504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e2e87b0-476b-40e9-873b-cae6a3e51cac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.459445 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.459428 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnk5\" (UniqueName: \"kubernetes.io/projected/4e2e87b0-476b-40e9-873b-cae6a3e51cac-kube-api-access-8cnk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.472430 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.472401 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:20.581562 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.581484 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:20.722890 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:20.722846 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z"] Jul 23 00:26:20.725098 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:20.725063 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2e87b0_476b_40e9_873b_cae6a3e51cac.slice/crio-304d2ee63c777f7eba5563c70bddc00eaae2d99d1dcee5f2e262376c54da5c85 WatchSource:0}: Error finding container 304d2ee63c777f7eba5563c70bddc00eaae2d99d1dcee5f2e262376c54da5c85: Status 404 returned error can't find the container with id 304d2ee63c777f7eba5563c70bddc00eaae2d99d1dcee5f2e262376c54da5c85 Jul 23 00:26:21.042045 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:21.042011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" event={"ID":"4e2e87b0-476b-40e9-873b-cae6a3e51cac","Type":"ContainerStarted","Data":"304d2ee63c777f7eba5563c70bddc00eaae2d99d1dcee5f2e262376c54da5c85"} Jul 23 00:26:21.048445 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:21.048421 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:23.301826 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.301784 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:23.706943 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.706875 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2rznc"] Jul 23 00:26:23.710315 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.710299 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:23.713713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.713696 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fngsw\"" Jul 23 00:26:23.733788 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.733764 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2rznc"] Jul 23 00:26:23.779580 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.779539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657nc\" (UniqueName: \"kubernetes.io/projected/d087e1fe-464c-4284-9ebd-7bdf767ad56b-kube-api-access-657nc\") pod \"authorino-operator-75cc8d5cb7-2rznc\" (UID: \"d087e1fe-464c-4284-9ebd-7bdf767ad56b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:23.881061 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.881030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657nc\" (UniqueName: \"kubernetes.io/projected/d087e1fe-464c-4284-9ebd-7bdf767ad56b-kube-api-access-657nc\") pod \"authorino-operator-75cc8d5cb7-2rznc\" (UID: \"d087e1fe-464c-4284-9ebd-7bdf767ad56b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:23.899226 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:23.899204 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-657nc\" (UniqueName: \"kubernetes.io/projected/d087e1fe-464c-4284-9ebd-7bdf767ad56b-kube-api-access-657nc\") pod \"authorino-operator-75cc8d5cb7-2rznc\" (UID: \"d087e1fe-464c-4284-9ebd-7bdf767ad56b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:24.021112 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:24.021088 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:24.177689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:24.177660 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2rznc"] Jul 23 00:26:24.179959 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:24.179933 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd087e1fe_464c_4284_9ebd_7bdf767ad56b.slice/crio-a15506befeb53c409c3e4a647293290fe90207a2ecabf1d5e47fb29bb64c5056 WatchSource:0}: Error finding container a15506befeb53c409c3e4a647293290fe90207a2ecabf1d5e47fb29bb64c5056: Status 404 returned error can't find the container with id a15506befeb53c409c3e4a647293290fe90207a2ecabf1d5e47fb29bb64c5056 Jul 23 00:26:24.543717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:24.543686 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:25.064897 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:25.064857 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" event={"ID":"d087e1fe-464c-4284-9ebd-7bdf767ad56b","Type":"ContainerStarted","Data":"a15506befeb53c409c3e4a647293290fe90207a2ecabf1d5e47fb29bb64c5056"} Jul 23 00:26:25.747955 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:25.747918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:26.186962 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.186884 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf"] Jul 23 00:26:26.190678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.190659 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:26.192970 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.192950 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rq8cb\"" Jul 23 00:26:26.201613 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.201554 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf"] Jul 23 00:26:26.304838 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.304808 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8bq\" (UniqueName: \"kubernetes.io/projected/6bb91562-bc6c-41e1-a434-0ba741451ab8-kube-api-access-2f8bq\") pod \"limitador-operator-controller-manager-859d7b78bd-w5llf\" (UID: \"6bb91562-bc6c-41e1-a434-0ba741451ab8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:26.405437 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.405406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8bq\" (UniqueName: \"kubernetes.io/projected/6bb91562-bc6c-41e1-a434-0ba741451ab8-kube-api-access-2f8bq\") pod \"limitador-operator-controller-manager-859d7b78bd-w5llf\" (UID: \"6bb91562-bc6c-41e1-a434-0ba741451ab8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:26.415177 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.415136 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8bq\" (UniqueName: \"kubernetes.io/projected/6bb91562-bc6c-41e1-a434-0ba741451ab8-kube-api-access-2f8bq\") pod \"limitador-operator-controller-manager-859d7b78bd-w5llf\" (UID: \"6bb91562-bc6c-41e1-a434-0ba741451ab8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:26.503722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.503676 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:26.547112 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:26.547056 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:27.790796 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:27.790767 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf"] Jul 23 00:26:27.791452 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:27.791426 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb91562_bc6c_41e1_a434_0ba741451ab8.slice/crio-cc8084858676a252fee5b5babccfb43fd36e827250b21f2d8758b666c7cf5a74 WatchSource:0}: Error finding container cc8084858676a252fee5b5babccfb43fd36e827250b21f2d8758b666c7cf5a74: Status 404 returned error can't find the container with id cc8084858676a252fee5b5babccfb43fd36e827250b21f2d8758b666c7cf5a74 Jul 23 00:26:27.803763 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:27.803734 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:27.905849 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:27.905760 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:28.005576 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.005539 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:28.080534 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.080495 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" event={"ID":"4e2e87b0-476b-40e9-873b-cae6a3e51cac","Type":"ContainerStarted","Data":"7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c"} Jul 23 00:26:28.080712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.080653 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:28.082074 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.082050 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" event={"ID":"d087e1fe-464c-4284-9ebd-7bdf767ad56b","Type":"ContainerStarted","Data":"eb03ce30ccd5c18baf69de08e9efce1829fb6dbc9b0668d12f97098c7bc90b45"} Jul 23 00:26:28.082207 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.082126 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:28.083259 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.083235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" event={"ID":"6bb91562-bc6c-41e1-a434-0ba741451ab8","Type":"ContainerStarted","Data":"cc8084858676a252fee5b5babccfb43fd36e827250b21f2d8758b666c7cf5a74"} Jul 23 00:26:28.099638 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.099602 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" podStartSLOduration=1.187923915 podStartE2EDuration="8.099589739s" totalImagesPullingTime="6.911665824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:28.098752609 +0000 UTC m=+417.395447600" watchObservedRunningTime="2026-07-23 00:26:28.099589739 +0000 UTC m=+417.396284942" Jul 23 00:26:28.116127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.116084 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" podStartSLOduration=1.678651422 podStartE2EDuration="5.11607113s" totalImagesPullingTime="3.437419708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:28.115142619 +0000 UTC m=+417.411837591" watchObservedRunningTime="2026-07-23 00:26:28.11607113 +0000 UTC m=+417.412766098" Jul 23 00:26:28.234978 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.234942 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:28.334438 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:28.334406 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:29.033583 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:29.033554 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcndc" Jul 23 00:26:30.094584 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:30.094554 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" event={"ID":"6bb91562-bc6c-41e1-a434-0ba741451ab8","Type":"ContainerStarted","Data":"8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57"} Jul 23 00:26:30.094941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:30.094614 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:30.113427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:30.113379 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" podStartSLOduration=2.009556977 podStartE2EDuration="4.113365341s" totalImagesPullingTime="2.103808364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:30.111627484 +0000 UTC m=+419.408322475" watchObservedRunningTime="2026-07-23 00:26:30.113365341 +0000 UTC m=+419.410060308" Jul 23 00:26:30.713892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:30.713848 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:31.101022 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:31.100989 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:31.471377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:31.471286 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:31.546707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:31.546673 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:38.202812 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:38.202781 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:38.304733 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:38.304692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:38.418231 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:38.418200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:39.090775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:39.090746 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:39.090937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:39.090924 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2rznc" Jul 23 00:26:40.849581 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:40.849538 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:41.031599 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.031551 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:41.102588 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.102450 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:41.124602 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.124577 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z"] Jul 23 00:26:41.124803 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.124781 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" containerName="manager" containerID="cri-o://7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c" gracePeriod=2 Jul 23 00:26:41.144403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.144364 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z"] Jul 23 00:26:41.155410 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:41.149481 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:41.167307 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.167280 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf"] Jul 23 00:26:41.167916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.167895 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" containerName="manager" Jul 23 00:26:41.167973 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.167921 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" containerName="manager" Jul 23 00:26:41.168021 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.168009 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" containerName="manager" Jul 23 00:26:41.171403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.171386 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.173328 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:41.173296 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:41.173587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.173306 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf"] Jul 23 00:26:41.173587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.173562 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" containerName="manager" containerID="cri-o://8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57" gracePeriod=2 Jul 23 00:26:41.183211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.183186 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf"] Jul 23 00:26:41.199183 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.199159 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf"] Jul 23 00:26:41.222273 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.222252 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77"] Jul 23 00:26:41.222669 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.222654 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" containerName="manager" Jul 23 00:26:41.222754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.222672 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" containerName="manager" Jul 23 00:26:41.222822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.222810 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" containerName="manager" Jul 23 00:26:41.225784 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.225767 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:41.234356 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.234323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d545e53f-6200-40ec-9dec-3b6a025d7af9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd2jf\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.234439 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.234402 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/d545e53f-6200-40ec-9dec-3b6a025d7af9-kube-api-access-nzf7h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd2jf\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.234526 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.234448 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kqd\" (UniqueName: \"kubernetes.io/projected/c26161d7-a9d3-4bb6-b82b-90d5b43bae72-kube-api-access-p6kqd\") pod \"limitador-operator-controller-manager-859d7b78bd-nzc77\" (UID: \"c26161d7-a9d3-4bb6-b82b-90d5b43bae72\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:41.261407 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.261384 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77"] Jul 23 00:26:41.264320 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:41.264289 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:41.269028 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:41.269002 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-w5llf\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:41.339148 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.339087 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d545e53f-6200-40ec-9dec-3b6a025d7af9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd2jf\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.339342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.339179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/d545e53f-6200-40ec-9dec-3b6a025d7af9-kube-api-access-nzf7h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd2jf\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.339342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.339229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kqd\" (UniqueName: \"kubernetes.io/projected/c26161d7-a9d3-4bb6-b82b-90d5b43bae72-kube-api-access-p6kqd\") pod \"limitador-operator-controller-manager-859d7b78bd-nzc77\" (UID: \"c26161d7-a9d3-4bb6-b82b-90d5b43bae72\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:41.339827 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.339803 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d545e53f-6200-40ec-9dec-3b6a025d7af9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd2jf\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.355782 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.355753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/d545e53f-6200-40ec-9dec-3b6a025d7af9-kube-api-access-nzf7h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd2jf\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.356049 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.356029 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kqd\" (UniqueName: \"kubernetes.io/projected/c26161d7-a9d3-4bb6-b82b-90d5b43bae72-kube-api-access-p6kqd\") pod \"limitador-operator-controller-manager-859d7b78bd-nzc77\" (UID: \"c26161d7-a9d3-4bb6-b82b-90d5b43bae72\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:41.410973 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:41.410935 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kgl6z\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:41.412632 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:41.412597 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-w5llf\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:41.418562 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.418541 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:41.426301 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.426283 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:41.540547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.540517 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e2e87b0-476b-40e9-873b-cae6a3e51cac-extensions-socket-volume\") pod \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " Jul 23 00:26:41.540720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.540557 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cnk5\" (UniqueName: \"kubernetes.io/projected/4e2e87b0-476b-40e9-873b-cae6a3e51cac-kube-api-access-8cnk5\") pod \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\" (UID: \"4e2e87b0-476b-40e9-873b-cae6a3e51cac\") " Jul 23 00:26:41.540720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.540581 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f8bq\" (UniqueName: \"kubernetes.io/projected/6bb91562-bc6c-41e1-a434-0ba741451ab8-kube-api-access-2f8bq\") pod \"6bb91562-bc6c-41e1-a434-0ba741451ab8\" (UID: \"6bb91562-bc6c-41e1-a434-0ba741451ab8\") " Jul 23 00:26:41.540903 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.540879 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2e87b0-476b-40e9-873b-cae6a3e51cac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4e2e87b0-476b-40e9-873b-cae6a3e51cac" (UID: "4e2e87b0-476b-40e9-873b-cae6a3e51cac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:26:41.542657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.542636 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2e87b0-476b-40e9-873b-cae6a3e51cac-kube-api-access-8cnk5" (OuterVolumeSpecName: "kube-api-access-8cnk5") pod "4e2e87b0-476b-40e9-873b-cae6a3e51cac" (UID: "4e2e87b0-476b-40e9-873b-cae6a3e51cac"). InnerVolumeSpecName "kube-api-access-8cnk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:26:41.542731 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.542657 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb91562-bc6c-41e1-a434-0ba741451ab8-kube-api-access-2f8bq" (OuterVolumeSpecName: "kube-api-access-2f8bq") pod "6bb91562-bc6c-41e1-a434-0ba741451ab8" (UID: "6bb91562-bc6c-41e1-a434-0ba741451ab8"). InnerVolumeSpecName "kube-api-access-2f8bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:26:41.557819 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.557797 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:41.564622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.564603 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:41.641272 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.641240 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e2e87b0-476b-40e9-873b-cae6a3e51cac-extensions-socket-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:26:41.641272 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.641273 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnk5\" (UniqueName: \"kubernetes.io/projected/4e2e87b0-476b-40e9-873b-cae6a3e51cac-kube-api-access-8cnk5\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:26:41.641511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.641292 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f8bq\" (UniqueName: \"kubernetes.io/projected/6bb91562-bc6c-41e1-a434-0ba741451ab8-kube-api-access-2f8bq\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:26:41.678951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.678918 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r"] Jul 23 00:26:41.685294 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.685271 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.734919 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.734891 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r"] Jul 23 00:26:41.741892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.741870 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bds\" (UniqueName: \"kubernetes.io/projected/83a9c8c6-65c7-467c-9965-fba02489d1a8-kube-api-access-j2bds\") pod \"kuadrant-operator-controller-manager-86878c769f-z6t2r\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.741987 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.741937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a9c8c6-65c7-467c-9965-fba02489d1a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z6t2r\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.743528 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.743505 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77"] Jul 23 00:26:41.744526 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:41.744503 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26161d7_a9d3_4bb6_b82b_90d5b43bae72.slice/crio-9e926f8e226bce89331a63838094b59f40e5a840bbcca9d1e8237b25a1247b43 WatchSource:0}: Error finding container 9e926f8e226bce89331a63838094b59f40e5a840bbcca9d1e8237b25a1247b43: Status 404 returned error can't find the container with id 9e926f8e226bce89331a63838094b59f40e5a840bbcca9d1e8237b25a1247b43 Jul 23 00:26:41.786262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.786233 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf"] Jul 23 00:26:41.788050 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:41.788026 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd545e53f_6200_40ec_9dec_3b6a025d7af9.slice/crio-26ee4b83107d40a4dd11ca87e73c8169441babe752be52461650ed48d058e351 WatchSource:0}: Error finding container 26ee4b83107d40a4dd11ca87e73c8169441babe752be52461650ed48d058e351: Status 404 returned error can't find the container with id 26ee4b83107d40a4dd11ca87e73c8169441babe752be52461650ed48d058e351 Jul 23 00:26:41.843107 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.842980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bds\" (UniqueName: \"kubernetes.io/projected/83a9c8c6-65c7-467c-9965-fba02489d1a8-kube-api-access-j2bds\") pod \"kuadrant-operator-controller-manager-86878c769f-z6t2r\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.843107 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.843036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a9c8c6-65c7-467c-9965-fba02489d1a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z6t2r\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.843361 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.843344 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a9c8c6-65c7-467c-9965-fba02489d1a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z6t2r\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.866367 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.866309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bds\" (UniqueName: \"kubernetes.io/projected/83a9c8c6-65c7-467c-9965-fba02489d1a8-kube-api-access-j2bds\") pod \"kuadrant-operator-controller-manager-86878c769f-z6t2r\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:41.997947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:41.997917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:42.162132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.162110 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r"] Jul 23 00:26:42.164548 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.164521 2585 generic.go:356] "Generic (PLEG): container finished" podID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" containerID="7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c" exitCode=0 Jul 23 00:26:42.164696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.164565 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kgl6z" Jul 23 00:26:42.164696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.164620 2585 scope.go:117] "RemoveContainer" containerID="7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c" Jul 23 00:26:42.165342 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:26:42.165288 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a9c8c6_65c7_467c_9965_fba02489d1a8.slice/crio-cc1716b1a4ec10186063a78042a0a14ad1f5358e1064fe62905d76ba9e5887c0 WatchSource:0}: Error finding container cc1716b1a4ec10186063a78042a0a14ad1f5358e1064fe62905d76ba9e5887c0: Status 404 returned error can't find the container with id cc1716b1a4ec10186063a78042a0a14ad1f5358e1064fe62905d76ba9e5887c0 Jul 23 00:26:42.166129 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.166108 2585 generic.go:356] "Generic (PLEG): container finished" podID="6bb91562-bc6c-41e1-a434-0ba741451ab8" containerID="8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57" exitCode=0 Jul 23 00:26:42.166217 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.166144 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w5llf" Jul 23 00:26:42.168189 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.168161 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" event={"ID":"c26161d7-a9d3-4bb6-b82b-90d5b43bae72","Type":"ContainerStarted","Data":"8e3524fbf0569f23319fc690aba798fb7d8cb58b0f8d18fda4d9c904cfdad1cc"} Jul 23 00:26:42.168291 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.168213 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" event={"ID":"c26161d7-a9d3-4bb6-b82b-90d5b43bae72","Type":"ContainerStarted","Data":"9e926f8e226bce89331a63838094b59f40e5a840bbcca9d1e8237b25a1247b43"} Jul 23 00:26:42.168919 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.168556 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:42.170347 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.170328 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" event={"ID":"d545e53f-6200-40ec-9dec-3b6a025d7af9","Type":"ContainerStarted","Data":"1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0"} Jul 23 00:26:42.170435 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.170369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" event={"ID":"d545e53f-6200-40ec-9dec-3b6a025d7af9","Type":"ContainerStarted","Data":"26ee4b83107d40a4dd11ca87e73c8169441babe752be52461650ed48d058e351"} Jul 23 00:26:42.174087 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.174045 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:42.177946 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.177924 2585 scope.go:117] "RemoveContainer" containerID="7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c" Jul 23 00:26:42.178197 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:42.178182 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c\": container with ID starting with 7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c not found: ID does not exist" containerID="7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c" Jul 23 00:26:42.178276 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.178207 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c"} err="failed to get container status \"7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c\": rpc error: code = NotFound desc = could not find container \"7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c\": container with ID starting with 7dc1f432ce266e85937e2d99910729cf699807bcdea147bfe3b14f172da3132c not found: ID does not exist" Jul 23 00:26:42.178276 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.178228 2585 scope.go:117] "RemoveContainer" containerID="8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57" Jul 23 00:26:42.186544 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.186526 2585 scope.go:117] "RemoveContainer" containerID="8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57" Jul 23 00:26:42.186831 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:42.186813 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57\": container with ID starting with 8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57 not found: ID does not exist" containerID="8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57" Jul 23 00:26:42.186876 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.186838 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57"} err="failed to get container status \"8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57\": rpc error: code = NotFound desc = could not find container \"8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57\": container with ID starting with 8d71b7dcd5de77879321db6e43b5e63abe3676f4cdd4a71ce336db7e15bbfa57 not found: ID does not exist" Jul 23 00:26:42.197355 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.197319 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" podStartSLOduration=1.197306558 podStartE2EDuration="1.197306558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:42.196647372 +0000 UTC m=+431.493342341" watchObservedRunningTime="2026-07-23 00:26:42.197306558 +0000 UTC m=+431.494001527" Jul 23 00:26:42.245542 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:42.245496 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" podStartSLOduration=1.245451761 podStartE2EDuration="1.245451761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:42.243323459 +0000 UTC m=+431.540018427" watchObservedRunningTime="2026-07-23 00:26:42.245451761 +0000 UTC m=+431.542146730" Jul 23 00:26:43.178525 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.178487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" event={"ID":"83a9c8c6-65c7-467c-9965-fba02489d1a8","Type":"ContainerStarted","Data":"fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f"} Jul 23 00:26:43.178972 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.178598 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:43.178972 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.178624 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" event={"ID":"83a9c8c6-65c7-467c-9965-fba02489d1a8","Type":"ContainerStarted","Data":"cc1716b1a4ec10186063a78042a0a14ad1f5358e1064fe62905d76ba9e5887c0"} Jul 23 00:26:43.209935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.209889 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" podStartSLOduration=2.209872835 podStartE2EDuration="2.209872835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:43.208836314 +0000 UTC m=+432.505531301" watchObservedRunningTime="2026-07-23 00:26:43.209872835 +0000 UTC m=+432.506567800" Jul 23 00:26:43.412544 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.412510 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2e87b0-476b-40e9-873b-cae6a3e51cac" path="/var/lib/kubelet/pods/4e2e87b0-476b-40e9-873b-cae6a3e51cac/volumes" Jul 23 00:26:43.412845 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.412832 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb91562-bc6c-41e1-a434-0ba741451ab8" path="/var/lib/kubelet/pods/6bb91562-bc6c-41e1-a434-0ba741451ab8/volumes" Jul 23 00:26:43.719756 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.719710 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:43.829942 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:43.829909 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:47.391304 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:47.391264 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:47.516153 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:47.516114 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:51.587871 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:51.587836 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:51.657265 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:51.657234 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:53.181918 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:53.181888 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:53.182316 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:53.181963 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nzc77" Jul 23 00:26:54.186980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.186907 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:26:54.249623 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.249592 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf"] Jul 23 00:26:54.249802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.249779 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" podUID="d545e53f-6200-40ec-9dec-3b6a025d7af9" containerName="manager" containerID="cri-o://1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0" gracePeriod=10 Jul 23 00:26:54.499058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.499033 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:54.549892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.549865 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d545e53f-6200-40ec-9dec-3b6a025d7af9-extensions-socket-volume\") pod \"d545e53f-6200-40ec-9dec-3b6a025d7af9\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " Jul 23 00:26:54.550043 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.549972 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/d545e53f-6200-40ec-9dec-3b6a025d7af9-kube-api-access-nzf7h\") pod \"d545e53f-6200-40ec-9dec-3b6a025d7af9\" (UID: \"d545e53f-6200-40ec-9dec-3b6a025d7af9\") " Jul 23 00:26:54.550271 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.550244 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d545e53f-6200-40ec-9dec-3b6a025d7af9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d545e53f-6200-40ec-9dec-3b6a025d7af9" (UID: "d545e53f-6200-40ec-9dec-3b6a025d7af9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:26:54.552050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.552029 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d545e53f-6200-40ec-9dec-3b6a025d7af9-kube-api-access-nzf7h" (OuterVolumeSpecName: "kube-api-access-nzf7h") pod "d545e53f-6200-40ec-9dec-3b6a025d7af9" (UID: "d545e53f-6200-40ec-9dec-3b6a025d7af9"). InnerVolumeSpecName "kube-api-access-nzf7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:26:54.650963 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.650931 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d545e53f-6200-40ec-9dec-3b6a025d7af9-extensions-socket-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:26:54.650963 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:54.650955 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/d545e53f-6200-40ec-9dec-3b6a025d7af9-kube-api-access-nzf7h\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:26:55.242565 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.242530 2585 generic.go:356] "Generic (PLEG): container finished" podID="d545e53f-6200-40ec-9dec-3b6a025d7af9" containerID="1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0" exitCode=0 Jul 23 00:26:55.243028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.242575 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" Jul 23 00:26:55.243028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.242616 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" event={"ID":"d545e53f-6200-40ec-9dec-3b6a025d7af9","Type":"ContainerDied","Data":"1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0"} Jul 23 00:26:55.243028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.242666 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf" event={"ID":"d545e53f-6200-40ec-9dec-3b6a025d7af9","Type":"ContainerDied","Data":"26ee4b83107d40a4dd11ca87e73c8169441babe752be52461650ed48d058e351"} Jul 23 00:26:55.243028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.242689 2585 scope.go:117] "RemoveContainer" containerID="1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0" Jul 23 00:26:55.252015 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.251843 2585 scope.go:117] "RemoveContainer" containerID="1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0" Jul 23 00:26:55.252112 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:26:55.252095 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0\": container with ID starting with 1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0 not found: ID does not exist" containerID="1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0" Jul 23 00:26:55.252154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.252122 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0"} err="failed to get container status \"1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0\": rpc error: code = NotFound desc = could not find container \"1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0\": container with ID starting with 1b8d6d5c12da409e29ea1c4a843d08b43282458233a215c14b004356b3af78f0 not found: ID does not exist" Jul 23 00:26:55.273011 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.272987 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf"] Jul 23 00:26:55.294218 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.294198 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd2jf"] Jul 23 00:26:55.411379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:26:55.411344 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d545e53f-6200-40ec-9dec-3b6a025d7af9" path="/var/lib/kubelet/pods/d545e53f-6200-40ec-9dec-3b6a025d7af9/volumes" Jul 23 00:27:01.501320 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:01.501280 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:01.601821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:01.601779 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:02.093320 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:02.093287 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:02.191970 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:02.191938 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:05.998767 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:05.998730 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:06.100009 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:06.099955 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:10.667237 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.667208 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z"] Jul 23 00:27:10.667640 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.667625 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d545e53f-6200-40ec-9dec-3b6a025d7af9" containerName="manager" Jul 23 00:27:10.667691 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.667643 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d545e53f-6200-40ec-9dec-3b6a025d7af9" containerName="manager" Jul 23 00:27:10.667729 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.667719 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d545e53f-6200-40ec-9dec-3b6a025d7af9" containerName="manager" Jul 23 00:27:10.672687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.672667 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.675383 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.675360 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tpvwm\"" Jul 23 00:27:10.682532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.682507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z"] Jul 23 00:27:10.706264 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.706238 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:10.792372 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792553 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792388 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792553 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fc3c40-0734-4a37-ad19-85cbbed76224-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792553 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792670 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792670 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792734 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792734 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792720 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8w28\" (UniqueName: \"kubernetes.io/projected/04fc3c40-0734-4a37-ad19-85cbbed76224-kube-api-access-l8w28\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.792834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.792766 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893287 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8w28\" (UniqueName: \"kubernetes.io/projected/04fc3c40-0734-4a37-ad19-85cbbed76224-kube-api-access-l8w28\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893356 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fc3c40-0734-4a37-ad19-85cbbed76224-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893419 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893743 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893743 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.893852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.893781 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.894139 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.894119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.894345 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.894319 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.895327 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.895303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fc3c40-0734-4a37-ad19-85cbbed76224-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.896485 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.896443 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.896589 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.896490 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.900727 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.900702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fc3c40-0734-4a37-ad19-85cbbed76224-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.901428 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.901408 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8w28\" (UniqueName: \"kubernetes.io/projected/04fc3c40-0734-4a37-ad19-85cbbed76224-kube-api-access-l8w28\") pod \"maas-default-gateway-openshift-default-56654dcf7-w7s5z\" (UID: \"04fc3c40-0734-4a37-ad19-85cbbed76224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:10.984731 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:10.984707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:11.065248 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.065208 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:11.121541 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.121477 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z"] Jul 23 00:27:11.124227 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:27:11.124198 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fc3c40_0734_4a37_ad19_85cbbed76224.slice/crio-eeadb8dc8f315f27f39afb9da2075a1f8d9c3d957ab66d028f1ba903d4040962 WatchSource:0}: Error finding container eeadb8dc8f315f27f39afb9da2075a1f8d9c3d957ab66d028f1ba903d4040962: Status 404 returned error can't find the container with id eeadb8dc8f315f27f39afb9da2075a1f8d9c3d957ab66d028f1ba903d4040962 Jul 23 00:27:11.126591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.126561 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:27:11.126760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.126624 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:27:11.126760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.126654 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:27:11.328824 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.328739 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" event={"ID":"04fc3c40-0734-4a37-ad19-85cbbed76224","Type":"ContainerStarted","Data":"33178e1169b719a7052c6066fb7ac6f300d8c66cd7868df34f46204a151ddfa7"} Jul 23 00:27:11.328824 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.328803 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" event={"ID":"04fc3c40-0734-4a37-ad19-85cbbed76224","Type":"ContainerStarted","Data":"eeadb8dc8f315f27f39afb9da2075a1f8d9c3d957ab66d028f1ba903d4040962"} Jul 23 00:27:11.349065 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.349011 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" podStartSLOduration=1.3489923099999999 podStartE2EDuration="1.34899231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:11.346653492 +0000 UTC m=+460.643348459" watchObservedRunningTime="2026-07-23 00:27:11.34899231 +0000 UTC m=+460.645687280" Jul 23 00:27:11.985321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:11.985288 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:12.612951 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:12.612918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:12.714260 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:12.714221 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:12.990208 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:12.990182 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:13.274749 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.274673 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-hkg42"] Jul 23 00:27:13.277527 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.277506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:13.279412 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.279396 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pslbj\"" Jul 23 00:27:13.286608 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.286584 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-hkg42"] Jul 23 00:27:13.308012 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.307987 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:13.337848 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.337825 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:13.338630 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.338611 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:27:13.389063 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.389030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:13.418625 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.418596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vhm\" (UniqueName: \"kubernetes.io/projected/fc0f9a43-583d-42e0-92dd-6f754c98820d-kube-api-access-64vhm\") pod \"authorino-58cd955675-hkg42\" (UID: \"fc0f9a43-583d-42e0-92dd-6f754c98820d\") " pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:13.519295 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.519255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vhm\" (UniqueName: \"kubernetes.io/projected/fc0f9a43-583d-42e0-92dd-6f754c98820d-kube-api-access-64vhm\") pod \"authorino-58cd955675-hkg42\" (UID: \"fc0f9a43-583d-42e0-92dd-6f754c98820d\") " pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:13.526902 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.526839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vhm\" (UniqueName: \"kubernetes.io/projected/fc0f9a43-583d-42e0-92dd-6f754c98820d-kube-api-access-64vhm\") pod \"authorino-58cd955675-hkg42\" (UID: \"fc0f9a43-583d-42e0-92dd-6f754c98820d\") " pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:13.587190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.587162 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:13.720300 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:13.720273 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-hkg42"] Jul 23 00:27:13.722160 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:27:13.722137 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0f9a43_583d_42e0_92dd_6f754c98820d.slice/crio-7f84d5b36980400becaef76186ba9be6fbf6f80f685b3d6e38b1c27296847935 WatchSource:0}: Error finding container 7f84d5b36980400becaef76186ba9be6fbf6f80f685b3d6e38b1c27296847935: Status 404 returned error can't find the container with id 7f84d5b36980400becaef76186ba9be6fbf6f80f685b3d6e38b1c27296847935 Jul 23 00:27:14.343672 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:14.343639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hkg42" event={"ID":"fc0f9a43-583d-42e0-92dd-6f754c98820d","Type":"ContainerStarted","Data":"7f84d5b36980400becaef76186ba9be6fbf6f80f685b3d6e38b1c27296847935"} Jul 23 00:27:17.362755 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:17.362720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hkg42" event={"ID":"fc0f9a43-583d-42e0-92dd-6f754c98820d","Type":"ContainerStarted","Data":"62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c"} Jul 23 00:27:17.376516 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:17.376452 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-hkg42" podStartSLOduration=1.194962703 podStartE2EDuration="4.376438192s" totalImagesPullingTime="3.181475489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:17.375450453 +0000 UTC m=+466.672145421" watchObservedRunningTime="2026-07-23 00:27:17.376438192 +0000 UTC m=+466.673133159" Jul 23 00:27:18.046516 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:18.046454 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-hkg42"] Jul 23 00:27:19.373122 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:19.373070 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-hkg42" podUID="fc0f9a43-583d-42e0-92dd-6f754c98820d" containerName="authorino" containerID="cri-o://62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c" gracePeriod=30 Jul 23 00:27:19.629132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:19.629072 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:19.777646 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:19.777613 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vhm\" (UniqueName: \"kubernetes.io/projected/fc0f9a43-583d-42e0-92dd-6f754c98820d-kube-api-access-64vhm\") pod \"fc0f9a43-583d-42e0-92dd-6f754c98820d\" (UID: \"fc0f9a43-583d-42e0-92dd-6f754c98820d\") " Jul 23 00:27:19.779738 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:19.779701 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0f9a43-583d-42e0-92dd-6f754c98820d-kube-api-access-64vhm" (OuterVolumeSpecName: "kube-api-access-64vhm") pod "fc0f9a43-583d-42e0-92dd-6f754c98820d" (UID: "fc0f9a43-583d-42e0-92dd-6f754c98820d"). InnerVolumeSpecName "kube-api-access-64vhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:27:19.878917 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:19.878888 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vhm\" (UniqueName: \"kubernetes.io/projected/fc0f9a43-583d-42e0-92dd-6f754c98820d-kube-api-access-64vhm\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:27:20.379043 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.379009 2585 generic.go:356] "Generic (PLEG): container finished" podID="fc0f9a43-583d-42e0-92dd-6f754c98820d" containerID="62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c" exitCode=0 Jul 23 00:27:20.379530 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.379062 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hkg42" Jul 23 00:27:20.379530 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.379089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hkg42" event={"ID":"fc0f9a43-583d-42e0-92dd-6f754c98820d","Type":"ContainerDied","Data":"62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c"} Jul 23 00:27:20.379530 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.379129 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hkg42" event={"ID":"fc0f9a43-583d-42e0-92dd-6f754c98820d","Type":"ContainerDied","Data":"7f84d5b36980400becaef76186ba9be6fbf6f80f685b3d6e38b1c27296847935"} Jul 23 00:27:20.379530 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.379147 2585 scope.go:117] "RemoveContainer" containerID="62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c" Jul 23 00:27:20.388333 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.388317 2585 scope.go:117] "RemoveContainer" containerID="62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c" Jul 23 00:27:20.388616 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:27:20.388598 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c\": container with ID starting with 62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c not found: ID does not exist" containerID="62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c" Jul 23 00:27:20.388680 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.388623 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c"} err="failed to get container status \"62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c\": rpc error: code = NotFound desc = could not find container \"62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c\": container with ID starting with 62cdcd5827b992be142f2e8d4c15237159ad63dc9ee19d3827c9f748c1143e4c not found: ID does not exist" Jul 23 00:27:20.399036 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.399011 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-hkg42"] Jul 23 00:27:20.402566 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:20.402539 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-hkg42"] Jul 23 00:27:21.410902 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:21.410865 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0f9a43-583d-42e0-92dd-6f754c98820d" path="/var/lib/kubelet/pods/fc0f9a43-583d-42e0-92dd-6f754c98820d/volumes" Jul 23 00:27:31.157082 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:31.157046 2585 scope.go:117] "RemoveContainer" containerID="07b5f010a2a5c47662c73933971842e1613a409feb746f44876ce4c84ad84b4f" Jul 23 00:27:33.742664 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:33.742621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:33.846207 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:33.846172 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.302869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:34.302838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:46.677050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.677010 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:46.781057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.781016 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:46.924088 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.924054 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sfbv"] Jul 23 00:27:46.924489 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.924457 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f9a43-583d-42e0-92dd-6f754c98820d" containerName="authorino" Jul 23 00:27:46.924489 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.924486 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f9a43-583d-42e0-92dd-6f754c98820d" containerName="authorino" Jul 23 00:27:46.924575 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.924564 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f9a43-583d-42e0-92dd-6f754c98820d" containerName="authorino" Jul 23 00:27:46.928169 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.928119 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:46.930315 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.930292 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pslbj\"" Jul 23 00:27:46.935824 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:46.935801 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sfbv"] Jul 23 00:27:47.015287 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.015253 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6248n\" (UniqueName: \"kubernetes.io/projected/5a258cf3-4401-46c5-96e3-ade0e58e5f06-kube-api-access-6248n\") pod \"authorino-5d4b67c446-4sfbv\" (UID: \"5a258cf3-4401-46c5-96e3-ade0e58e5f06\") " pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:47.115844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.115809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6248n\" (UniqueName: \"kubernetes.io/projected/5a258cf3-4401-46c5-96e3-ade0e58e5f06-kube-api-access-6248n\") pod \"authorino-5d4b67c446-4sfbv\" (UID: \"5a258cf3-4401-46c5-96e3-ade0e58e5f06\") " pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:47.123296 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.123262 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6248n\" (UniqueName: \"kubernetes.io/projected/5a258cf3-4401-46c5-96e3-ade0e58e5f06-kube-api-access-6248n\") pod \"authorino-5d4b67c446-4sfbv\" (UID: \"5a258cf3-4401-46c5-96e3-ade0e58e5f06\") " pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:47.190938 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.190872 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sfbv"] Jul 23 00:27:47.191126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.191115 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:47.218521 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.218496 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f9f9c877-mx9b6"] Jul 23 00:27:47.221356 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.221336 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.223656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.223634 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 00:27:47.232273 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.232249 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9f9c877-mx9b6"] Jul 23 00:27:47.317704 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.317674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19cabda7-ccc0-4797-b27f-a50433207d15-tls-cert\") pod \"authorino-5f9f9c877-mx9b6\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.317850 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.317713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl57j\" (UniqueName: \"kubernetes.io/projected/19cabda7-ccc0-4797-b27f-a50433207d15-kube-api-access-tl57j\") pod \"authorino-5f9f9c877-mx9b6\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.328911 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.328890 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sfbv"] Jul 23 00:27:47.330938 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:27:47.330912 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a258cf3_4401_46c5_96e3_ade0e58e5f06.slice/crio-b83b0dd1f4995d8aef240ad86d218c26563d275ea4824efdf9215538bfc5248e WatchSource:0}: Error finding container b83b0dd1f4995d8aef240ad86d218c26563d275ea4824efdf9215538bfc5248e: Status 404 returned error can't find the container with id b83b0dd1f4995d8aef240ad86d218c26563d275ea4824efdf9215538bfc5248e Jul 23 00:27:47.418528 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.418501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19cabda7-ccc0-4797-b27f-a50433207d15-tls-cert\") pod \"authorino-5f9f9c877-mx9b6\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.418673 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.418536 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl57j\" (UniqueName: \"kubernetes.io/projected/19cabda7-ccc0-4797-b27f-a50433207d15-kube-api-access-tl57j\") pod \"authorino-5f9f9c877-mx9b6\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.420815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.420795 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19cabda7-ccc0-4797-b27f-a50433207d15-tls-cert\") pod \"authorino-5f9f9c877-mx9b6\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.425596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.425572 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl57j\" (UniqueName: \"kubernetes.io/projected/19cabda7-ccc0-4797-b27f-a50433207d15-kube-api-access-tl57j\") pod \"authorino-5f9f9c877-mx9b6\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.507930 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.507901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" event={"ID":"5a258cf3-4401-46c5-96e3-ade0e58e5f06","Type":"ContainerStarted","Data":"b83b0dd1f4995d8aef240ad86d218c26563d275ea4824efdf9215538bfc5248e"} Jul 23 00:27:47.533238 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.533215 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:27:47.665764 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:47.665738 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9f9c877-mx9b6"] Jul 23 00:27:47.666328 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:27:47.666306 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19cabda7_ccc0_4797_b27f_a50433207d15.slice/crio-cd425a3cc0d21eb0354b9553675886dfeb7f668e2868ca69e3eec3cec6fec0a8 WatchSource:0}: Error finding container cd425a3cc0d21eb0354b9553675886dfeb7f668e2868ca69e3eec3cec6fec0a8: Status 404 returned error can't find the container with id cd425a3cc0d21eb0354b9553675886dfeb7f668e2868ca69e3eec3cec6fec0a8 Jul 23 00:27:48.514706 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.514666 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" event={"ID":"5a258cf3-4401-46c5-96e3-ade0e58e5f06","Type":"ContainerStarted","Data":"f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7"} Jul 23 00:27:48.514706 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.514681 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" podUID="5a258cf3-4401-46c5-96e3-ade0e58e5f06" containerName="authorino" containerID="cri-o://f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7" gracePeriod=30 Jul 23 00:27:48.516105 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.516077 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" event={"ID":"19cabda7-ccc0-4797-b27f-a50433207d15","Type":"ContainerStarted","Data":"24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839"} Jul 23 00:27:48.516233 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.516114 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" event={"ID":"19cabda7-ccc0-4797-b27f-a50433207d15","Type":"ContainerStarted","Data":"cd425a3cc0d21eb0354b9553675886dfeb7f668e2868ca69e3eec3cec6fec0a8"} Jul 23 00:27:48.529956 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.529919 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" podStartSLOduration=2.247846318 podStartE2EDuration="2.529905748s" totalImagesPullingTime="282.05943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:48.528403418 +0000 UTC m=+497.825098386" watchObservedRunningTime="2026-07-23 00:27:48.529905748 +0000 UTC m=+497.826600716" Jul 23 00:27:48.543160 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.543118 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" podStartSLOduration=1.20383398 podStartE2EDuration="1.543106854s" totalImagesPullingTime="339.272874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:48.541119261 +0000 UTC m=+497.837814258" watchObservedRunningTime="2026-07-23 00:27:48.543106854 +0000 UTC m=+497.839801821" Jul 23 00:27:48.767067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.767012 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:48.831448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.831406 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6248n\" (UniqueName: \"kubernetes.io/projected/5a258cf3-4401-46c5-96e3-ade0e58e5f06-kube-api-access-6248n\") pod \"5a258cf3-4401-46c5-96e3-ade0e58e5f06\" (UID: \"5a258cf3-4401-46c5-96e3-ade0e58e5f06\") " Jul 23 00:27:48.833569 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.833536 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a258cf3-4401-46c5-96e3-ade0e58e5f06-kube-api-access-6248n" (OuterVolumeSpecName: "kube-api-access-6248n") pod "5a258cf3-4401-46c5-96e3-ade0e58e5f06" (UID: "5a258cf3-4401-46c5-96e3-ade0e58e5f06"). InnerVolumeSpecName "kube-api-access-6248n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:27:48.932738 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:48.932703 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6248n\" (UniqueName: \"kubernetes.io/projected/5a258cf3-4401-46c5-96e3-ade0e58e5f06-kube-api-access-6248n\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:27:49.522343 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.522309 2585 generic.go:356] "Generic (PLEG): container finished" podID="5a258cf3-4401-46c5-96e3-ade0e58e5f06" containerID="f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7" exitCode=0 Jul 23 00:27:49.522794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.522352 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" Jul 23 00:27:49.522794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.522396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" event={"ID":"5a258cf3-4401-46c5-96e3-ade0e58e5f06","Type":"ContainerDied","Data":"f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7"} Jul 23 00:27:49.522794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.522442 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sfbv" event={"ID":"5a258cf3-4401-46c5-96e3-ade0e58e5f06","Type":"ContainerDied","Data":"b83b0dd1f4995d8aef240ad86d218c26563d275ea4824efdf9215538bfc5248e"} Jul 23 00:27:49.522794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.522477 2585 scope.go:117] "RemoveContainer" containerID="f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7" Jul 23 00:27:49.531332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.531315 2585 scope.go:117] "RemoveContainer" containerID="f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7" Jul 23 00:27:49.531612 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:27:49.531595 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7\": container with ID starting with f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7 not found: ID does not exist" containerID="f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7" Jul 23 00:27:49.531676 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.531620 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7"} err="failed to get container status \"f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7\": rpc error: code = NotFound desc = could not find container \"f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7\": container with ID starting with f71e7694b709bbcba9c6c16e6ba10e3e4b0ae37d8889130ede8c0efaf9578ac7 not found: ID does not exist" Jul 23 00:27:49.538626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.538600 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sfbv"] Jul 23 00:27:49.542307 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:49.542287 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sfbv"] Jul 23 00:27:51.411550 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:51.411521 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a258cf3-4401-46c5-96e3-ade0e58e5f06" path="/var/lib/kubelet/pods/5a258cf3-4401-46c5-96e3-ade0e58e5f06/volumes" Jul 23 00:27:53.394890 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:53.394857 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:53.493858 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:53.493821 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:53.582837 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:53.582803 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:56.615828 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:56.615770 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:56.718389 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:27:56.718348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:29.279140 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:29.279037 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:29.376830 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:29.376788 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:29.744451 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:29.744409 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:30.319223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:30.319187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:30.425485 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:30.425426 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:30.560070 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:30.560032 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:38.773253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.773219 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fc3c40-0734-4a37-ad19-85cbbed76224" containerID="33178e1169b719a7052c6066fb7ac6f300d8c66cd7868df34f46204a151ddfa7" exitCode=0 Jul 23 00:28:38.773705 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.773301 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" event={"ID":"04fc3c40-0734-4a37-ad19-85cbbed76224","Type":"ContainerDied","Data":"33178e1169b719a7052c6066fb7ac6f300d8c66cd7868df34f46204a151ddfa7"} Jul 23 00:28:38.773750 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.773705 2585 scope.go:117] "RemoveContainer" containerID="33178e1169b719a7052c6066fb7ac6f300d8c66cd7868df34f46204a151ddfa7" Jul 23 00:28:38.774703 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.774676 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:28:38.774772 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.774743 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:28:38.774812 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.774783 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:28:38.966384 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:38.966345 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:39.779655 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:39.779621 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" event={"ID":"04fc3c40-0734-4a37-ad19-85cbbed76224","Type":"ContainerStarted","Data":"da2d92684a6c64e2bfbd147b520f41b045b1896be8a3bb5ff2417998a13684c7"} Jul 23 00:28:39.984975 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:39.984939 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:28:39.986062 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:39.986039 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:28:39.986136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:39.986088 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" podUID="04fc3c40-0734-4a37-ad19-85cbbed76224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 23 00:28:40.656215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:40.656173 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:40.759428 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:40.759377 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:40.986061 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:40.986027 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:28:40.986430 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:40.986093 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" podUID="04fc3c40-0734-4a37-ad19-85cbbed76224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 23 00:28:41.985384 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:41.985346 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:28:41.985613 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:41.985407 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" podUID="04fc3c40-0734-4a37-ad19-85cbbed76224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 23 00:28:42.985105 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:42.985060 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-w7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:28:42.985575 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:42.985137 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" podUID="04fc3c40-0734-4a37-ad19-85cbbed76224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 23 00:28:43.338454 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:43.338358 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:28:43.989756 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:43.989724 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:28:43.990760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:43.990742 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-w7s5z" Jul 23 00:28:49.189363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:28:49.189325 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:00.061723 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.061683 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj"] Jul 23 00:29:00.062143 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.062128 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a258cf3-4401-46c5-96e3-ade0e58e5f06" containerName="authorino" Jul 23 00:29:00.062190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.062146 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a258cf3-4401-46c5-96e3-ade0e58e5f06" containerName="authorino" Jul 23 00:29:00.062225 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.062219 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a258cf3-4401-46c5-96e3-ade0e58e5f06" containerName="authorino" Jul 23 00:29:00.065301 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.065286 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.067631 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.067611 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 00:29:00.068254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.068237 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 00:29:00.068353 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.068277 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4vnxc\"" Jul 23 00:29:00.068353 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.068304 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 23 00:29:00.076354 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.076331 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj"] Jul 23 00:29:00.139167 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.139131 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.139332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.139176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rplr6\" (UniqueName: \"kubernetes.io/projected/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-kube-api-access-rplr6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.139332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.139215 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.139332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.139233 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.139332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.139272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.139536 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.139342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.151396 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.151363 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:00.240190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240193 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240411 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240241 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240411 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240411 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240602 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240412 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rplr6\" (UniqueName: \"kubernetes.io/projected/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-kube-api-access-rplr6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240667 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.240910 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.240880 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.242885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.242832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.243011 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.242970 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.248621 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.248595 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rplr6\" (UniqueName: \"kubernetes.io/projected/9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6-kube-api-access-rplr6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cwzvj\" (UID: \"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.376778 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.376689 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:00.536108 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.536078 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj"] Jul 23 00:29:00.538962 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:29:00.538925 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d696b5f_f8f1_44f3_8fae_3fd32cecc6c6.slice/crio-702cc7933446d2abccbff31b24967103d81fb92ea007b663284da8799ed66f47 WatchSource:0}: Error finding container 702cc7933446d2abccbff31b24967103d81fb92ea007b663284da8799ed66f47: Status 404 returned error can't find the container with id 702cc7933446d2abccbff31b24967103d81fb92ea007b663284da8799ed66f47 Jul 23 00:29:00.907945 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:00.907906 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" event={"ID":"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6","Type":"ContainerStarted","Data":"702cc7933446d2abccbff31b24967103d81fb92ea007b663284da8799ed66f47"} Jul 23 00:29:01.914867 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:01.914832 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" event={"ID":"9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6","Type":"ContainerStarted","Data":"12307644301a9bde335a80038e43114636c3b56db6f96da8b57b1922f743530c"} Jul 23 00:29:01.931725 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:01.931620 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" podStartSLOduration=0.791801313 podStartE2EDuration="1.931603721s" totalImagesPullingTime="1.139802408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:29:01.929888927 +0000 UTC m=+571.226583897" watchObservedRunningTime="2026-07-23 00:29:01.931603721 +0000 UTC m=+571.228298690" Jul 23 00:29:08.885110 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:08.885078 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:10.377816 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:10.377772 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:10.378215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:10.377837 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:10.391075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:10.391046 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:10.977833 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:10.977804 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cwzvj" Jul 23 00:29:28.198830 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:28.198790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:31.148710 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:31.148678 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:29:31.152504 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:31.152483 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:29:33.685758 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:33.685714 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:35.578890 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:29:35.578855 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:00.139182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.139102 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm"] Jul 23 00:30:00.142620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.142603 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" Jul 23 00:30:00.144490 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.144451 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:30:00.148708 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.148683 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm"] Jul 23 00:30:00.154130 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.154111 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" Jul 23 00:30:00.495527 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.495492 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm"] Jul 23 00:30:00.499312 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:30:00.499281 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c31eb0_6b2a_4dc8_8dcc_341fecff97d5.slice/crio-390e5e1bcd4dca2faf3dc45a7695056fbe2b9d937afbba514a5a93708470fdfb WatchSource:0}: Error finding container 390e5e1bcd4dca2faf3dc45a7695056fbe2b9d937afbba514a5a93708470fdfb: Status 404 returned error can't find the container with id 390e5e1bcd4dca2faf3dc45a7695056fbe2b9d937afbba514a5a93708470fdfb Jul 23 00:30:00.501013 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:00.500994 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:30:01.226777 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:01.226745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" event={"ID":"16c31eb0-6b2a-4dc8-8dcc-341fecff97d5","Type":"ContainerStarted","Data":"390e5e1bcd4dca2faf3dc45a7695056fbe2b9d937afbba514a5a93708470fdfb"} Jul 23 00:30:04.245334 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:04.245297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" event={"ID":"16c31eb0-6b2a-4dc8-8dcc-341fecff97d5","Type":"ContainerStarted","Data":"8bf37bdfd37a2c1f58a88847fef3fd97740d6a0ef84b46e9b0fc51306f784518"} Jul 23 00:30:04.260651 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:04.260544 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" podStartSLOduration=1.036588248 podStartE2EDuration="4.260528499s" totalImagesPullingTime="3.223940251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:30:04.260113766 +0000 UTC m=+633.556808769" watchObservedRunningTime="2026-07-23 00:30:04.260528499 +0000 UTC m=+633.557223467" Jul 23 00:30:05.251801 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:05.251769 2585 generic.go:356] "Generic (PLEG): container finished" podID="16c31eb0-6b2a-4dc8-8dcc-341fecff97d5" containerID="8bf37bdfd37a2c1f58a88847fef3fd97740d6a0ef84b46e9b0fc51306f784518" exitCode=0 Jul 23 00:30:05.252196 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:05.251859 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" event={"ID":"16c31eb0-6b2a-4dc8-8dcc-341fecff97d5","Type":"ContainerDied","Data":"8bf37bdfd37a2c1f58a88847fef3fd97740d6a0ef84b46e9b0fc51306f784518"} Jul 23 00:30:06.395246 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:06.395222 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" Jul 23 00:30:07.266638 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:07.266606 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" event={"ID":"16c31eb0-6b2a-4dc8-8dcc-341fecff97d5","Type":"ContainerDied","Data":"390e5e1bcd4dca2faf3dc45a7695056fbe2b9d937afbba514a5a93708470fdfb"} Jul 23 00:30:07.266812 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:07.266652 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="390e5e1bcd4dca2faf3dc45a7695056fbe2b9d937afbba514a5a93708470fdfb" Jul 23 00:30:07.266812 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:07.266620 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-nf9jm" Jul 23 00:30:17.089130 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:17.089094 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:19.541675 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:19.541625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:25.890436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:25.890400 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:33.915774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:33.915725 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:45.494762 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:45.494707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:52.551213 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:30:52.551172 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:12.077678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:12.077642 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:12.153067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:12.153030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:13.261624 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:13.261584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:13.360003 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:13.359959 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:14.054037 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.053998 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9f9c877-mx9b6"] Jul 23 00:31:14.054215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.054193 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" podUID="19cabda7-ccc0-4797-b27f-a50433207d15" containerName="authorino" containerID="cri-o://24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839" gracePeriod=30 Jul 23 00:31:14.303910 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.303885 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:31:14.387608 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.387530 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl57j\" (UniqueName: \"kubernetes.io/projected/19cabda7-ccc0-4797-b27f-a50433207d15-kube-api-access-tl57j\") pod \"19cabda7-ccc0-4797-b27f-a50433207d15\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " Jul 23 00:31:14.387608 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.387577 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19cabda7-ccc0-4797-b27f-a50433207d15-tls-cert\") pod \"19cabda7-ccc0-4797-b27f-a50433207d15\" (UID: \"19cabda7-ccc0-4797-b27f-a50433207d15\") " Jul 23 00:31:14.389708 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.389678 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cabda7-ccc0-4797-b27f-a50433207d15-kube-api-access-tl57j" (OuterVolumeSpecName: "kube-api-access-tl57j") pod "19cabda7-ccc0-4797-b27f-a50433207d15" (UID: "19cabda7-ccc0-4797-b27f-a50433207d15"). InnerVolumeSpecName "kube-api-access-tl57j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:31:14.398271 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.398248 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cabda7-ccc0-4797-b27f-a50433207d15-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "19cabda7-ccc0-4797-b27f-a50433207d15" (UID: "19cabda7-ccc0-4797-b27f-a50433207d15"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:31:14.489120 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.489080 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl57j\" (UniqueName: \"kubernetes.io/projected/19cabda7-ccc0-4797-b27f-a50433207d15-kube-api-access-tl57j\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:31:14.489120 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.489116 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19cabda7-ccc0-4797-b27f-a50433207d15-tls-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:31:14.594712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.594677 2585 generic.go:356] "Generic (PLEG): container finished" podID="19cabda7-ccc0-4797-b27f-a50433207d15" containerID="24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839" exitCode=0 Jul 23 00:31:14.594860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.594718 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" Jul 23 00:31:14.594860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.594745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" event={"ID":"19cabda7-ccc0-4797-b27f-a50433207d15","Type":"ContainerDied","Data":"24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839"} Jul 23 00:31:14.594860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.594784 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9f9c877-mx9b6" event={"ID":"19cabda7-ccc0-4797-b27f-a50433207d15","Type":"ContainerDied","Data":"cd425a3cc0d21eb0354b9553675886dfeb7f668e2868ca69e3eec3cec6fec0a8"} Jul 23 00:31:14.594860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.594804 2585 scope.go:117] "RemoveContainer" containerID="24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839" Jul 23 00:31:14.604578 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.604554 2585 scope.go:117] "RemoveContainer" containerID="24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839" Jul 23 00:31:14.604831 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:31:14.604814 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839\": container with ID starting with 24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839 not found: ID does not exist" containerID="24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839" Jul 23 00:31:14.604878 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.604838 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839"} err="failed to get container status \"24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839\": rpc error: code = NotFound desc = could not find container \"24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839\": container with ID starting with 24664bfe27873fd54c9374bfaac89154950b06f252c86fe967e7ebd540420839 not found: ID does not exist" Jul 23 00:31:14.617344 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.617321 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9f9c877-mx9b6"] Jul 23 00:31:14.621199 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:14.621177 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f9f9c877-mx9b6"] Jul 23 00:31:15.411785 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:31:15.411755 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cabda7-ccc0-4797-b27f-a50433207d15" path="/var/lib/kubelet/pods/19cabda7-ccc0-4797-b27f-a50433207d15/volumes" Jul 23 00:32:00.407981 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:32:00.407941 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:18.604620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:33:18.604582 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:19.862318 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:33:19.862271 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:55.677687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:33:55.677646 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:55.777011 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:33:55.776978 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:55.897024 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:33:55.896986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:01.464867 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:01.464831 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:01.531985 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:01.531954 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:01.611204 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:01.611173 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:01.710267 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:01.710230 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:01.850697 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:01.850665 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:07.590184 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:07.590149 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:07.661967 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:07.661926 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:31.183983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:31.183904 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:34:31.184925 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:31.184901 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:34:54.591673 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:54.591633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:54.646930 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:34:54.646896 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:10.176548 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:35:10.176511 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:16.739330 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:36:16.739295 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:58.128045 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:37:58.127964 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r"] Jul 23 00:37:58.128534 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:37:58.128179 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" podUID="83a9c8c6-65c7-467c-9965-fba02489d1a8" containerName="manager" containerID="cri-o://fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f" gracePeriod=10 Jul 23 00:38:00.187540 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.187511 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:38:00.270095 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.270020 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2bds\" (UniqueName: \"kubernetes.io/projected/83a9c8c6-65c7-467c-9965-fba02489d1a8-kube-api-access-j2bds\") pod \"83a9c8c6-65c7-467c-9965-fba02489d1a8\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " Jul 23 00:38:00.270233 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.270134 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a9c8c6-65c7-467c-9965-fba02489d1a8-extensions-socket-volume\") pod \"83a9c8c6-65c7-467c-9965-fba02489d1a8\" (UID: \"83a9c8c6-65c7-467c-9965-fba02489d1a8\") " Jul 23 00:38:00.270516 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.270487 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a9c8c6-65c7-467c-9965-fba02489d1a8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "83a9c8c6-65c7-467c-9965-fba02489d1a8" (UID: "83a9c8c6-65c7-467c-9965-fba02489d1a8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:38:00.272091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.272066 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a9c8c6-65c7-467c-9965-fba02489d1a8-kube-api-access-j2bds" (OuterVolumeSpecName: "kube-api-access-j2bds") pod "83a9c8c6-65c7-467c-9965-fba02489d1a8" (UID: "83a9c8c6-65c7-467c-9965-fba02489d1a8"). InnerVolumeSpecName "kube-api-access-j2bds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:38:00.371696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.371659 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a9c8c6-65c7-467c-9965-fba02489d1a8-extensions-socket-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:38:00.371696 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.371688 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bds\" (UniqueName: \"kubernetes.io/projected/83a9c8c6-65c7-467c-9965-fba02489d1a8-kube-api-access-j2bds\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:38:00.638359 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.638267 2585 generic.go:356] "Generic (PLEG): container finished" podID="83a9c8c6-65c7-467c-9965-fba02489d1a8" containerID="fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f" exitCode=0 Jul 23 00:38:00.638359 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.638307 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" Jul 23 00:38:00.638620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.638357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" event={"ID":"83a9c8c6-65c7-467c-9965-fba02489d1a8","Type":"ContainerDied","Data":"fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f"} Jul 23 00:38:00.638620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.638413 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r" event={"ID":"83a9c8c6-65c7-467c-9965-fba02489d1a8","Type":"ContainerDied","Data":"cc1716b1a4ec10186063a78042a0a14ad1f5358e1064fe62905d76ba9e5887c0"} Jul 23 00:38:00.638620 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.638444 2585 scope.go:117] "RemoveContainer" containerID="fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f" Jul 23 00:38:00.648687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.648668 2585 scope.go:117] "RemoveContainer" containerID="fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f" Jul 23 00:38:00.648951 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:38:00.648931 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f\": container with ID starting with fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f not found: ID does not exist" containerID="fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f" Jul 23 00:38:00.648998 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.648966 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f"} err="failed to get container status \"fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f\": rpc error: code = NotFound desc = could not find container \"fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f\": container with ID starting with fa8129dc421691dde5318a90b63e13761ee298f7ffbc1c2b1eb61dbebc34c57f not found: ID does not exist" Jul 23 00:38:00.659650 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.659629 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r"] Jul 23 00:38:00.663199 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:00.663182 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z6t2r"] Jul 23 00:38:01.418309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:01.418274 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a9c8c6-65c7-467c-9965-fba02489d1a8" path="/var/lib/kubelet/pods/83a9c8c6-65c7-467c-9965-fba02489d1a8/volumes" Jul 23 00:38:54.445917 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:54.445842 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:38:59.393692 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.393654 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj"] Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394034 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cabda7-ccc0-4797-b27f-a50433207d15" containerName="authorino" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394044 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cabda7-ccc0-4797-b27f-a50433207d15" containerName="authorino" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394058 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c31eb0-6b2a-4dc8-8dcc-341fecff97d5" containerName="cleanup" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394063 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c31eb0-6b2a-4dc8-8dcc-341fecff97d5" containerName="cleanup" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394071 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a9c8c6-65c7-467c-9965-fba02489d1a8" containerName="manager" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394076 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a9c8c6-65c7-467c-9965-fba02489d1a8" containerName="manager" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394140 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c31eb0-6b2a-4dc8-8dcc-341fecff97d5" containerName="cleanup" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394150 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a9c8c6-65c7-467c-9965-fba02489d1a8" containerName="manager" Jul 23 00:38:59.396016 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.394156 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cabda7-ccc0-4797-b27f-a50433207d15" containerName="authorino" Jul 23 00:38:59.397013 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.396997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.398954 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.398934 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2tgzd\"" Jul 23 00:38:59.412848 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.412819 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj"] Jul 23 00:38:59.461169 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.461137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6g6l\" (UniqueName: \"kubernetes.io/projected/381f3239-c898-46f2-b4e9-84716f61d46d-kube-api-access-f6g6l\") pod \"kuadrant-operator-controller-manager-86878c769f-vsbdj\" (UID: \"381f3239-c898-46f2-b4e9-84716f61d46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.461327 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.461177 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/381f3239-c898-46f2-b4e9-84716f61d46d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vsbdj\" (UID: \"381f3239-c898-46f2-b4e9-84716f61d46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.561837 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.561784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/381f3239-c898-46f2-b4e9-84716f61d46d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vsbdj\" (UID: \"381f3239-c898-46f2-b4e9-84716f61d46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.562025 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.561913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6g6l\" (UniqueName: \"kubernetes.io/projected/381f3239-c898-46f2-b4e9-84716f61d46d-kube-api-access-f6g6l\") pod \"kuadrant-operator-controller-manager-86878c769f-vsbdj\" (UID: \"381f3239-c898-46f2-b4e9-84716f61d46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.562196 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.562176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/381f3239-c898-46f2-b4e9-84716f61d46d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vsbdj\" (UID: \"381f3239-c898-46f2-b4e9-84716f61d46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.569912 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.569885 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6g6l\" (UniqueName: \"kubernetes.io/projected/381f3239-c898-46f2-b4e9-84716f61d46d-kube-api-access-f6g6l\") pod \"kuadrant-operator-controller-manager-86878c769f-vsbdj\" (UID: \"381f3239-c898-46f2-b4e9-84716f61d46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.708186 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.708092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.842867 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.842817 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj"] Jul 23 00:38:59.845502 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:38:59.845444 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381f3239_c898_46f2_b4e9_84716f61d46d.slice/crio-cbe91f274362c13ae9a3f4d28a411a7ceb5f99d428c90090c40632b61239c9d9 WatchSource:0}: Error finding container cbe91f274362c13ae9a3f4d28a411a7ceb5f99d428c90090c40632b61239c9d9: Status 404 returned error can't find the container with id cbe91f274362c13ae9a3f4d28a411a7ceb5f99d428c90090c40632b61239c9d9 Jul 23 00:38:59.931422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.931380 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" event={"ID":"381f3239-c898-46f2-b4e9-84716f61d46d","Type":"ContainerStarted","Data":"2dc462430e5799b1c2becbbe7bb890b634c22c7dbe107937860a5fb1be33f5e8"} Jul 23 00:38:59.931597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.931445 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" event={"ID":"381f3239-c898-46f2-b4e9-84716f61d46d","Type":"ContainerStarted","Data":"cbe91f274362c13ae9a3f4d28a411a7ceb5f99d428c90090c40632b61239c9d9"} Jul 23 00:38:59.931597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.931517 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:38:59.950375 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:38:59.950317 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" podStartSLOduration=0.950281615 podStartE2EDuration="950.281615ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:38:59.94995728 +0000 UTC m=+1169.246652247" watchObservedRunningTime="2026-07-23 00:38:59.950281615 +0000 UTC m=+1169.246976595" Jul 23 00:39:10.938671 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:39:10.938638 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vsbdj" Jul 23 00:39:31.216430 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:39:31.216400 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:39:31.218030 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:39:31.218007 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:41:03.536783 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:41:03.536704 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:50.888957 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:41:50.888919 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:58.289083 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:41:58.288992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:32.721673 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:42:32.721641 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:33.312688 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:42:33.312652 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:18.335820 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:43:18.335787 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:38.497476 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:43:38.497386 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:59.331707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:43:59.331671 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:31.248874 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:44:31.248843 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:44:31.251348 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:44:31.250245 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:45:00.145546 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.145429 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp"] Jul 23 00:45:00.149241 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.149215 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" Jul 23 00:45:00.151323 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.151298 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:45:00.158666 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.158650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" Jul 23 00:45:00.166243 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.166218 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp"] Jul 23 00:45:00.299879 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.299852 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp"] Jul 23 00:45:00.300832 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:45:00.300808 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5113eabc_e5b6_4685_8d3f_e860e31befdd.slice/crio-825508ef9ba576ea4eb20fbf162b8e83500b22f7e6e675ccb54b25a92f56b302 WatchSource:0}: Error finding container 825508ef9ba576ea4eb20fbf162b8e83500b22f7e6e675ccb54b25a92f56b302: Status 404 returned error can't find the container with id 825508ef9ba576ea4eb20fbf162b8e83500b22f7e6e675ccb54b25a92f56b302 Jul 23 00:45:00.755535 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.755497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" event={"ID":"5113eabc-e5b6-4685-8d3f-e860e31befdd","Type":"ContainerStarted","Data":"faae67a06c9ba1a959830e6e8870ac1ae67722f2ee6add28d25d3628f8821846"} Jul 23 00:45:00.755716 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.755549 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" event={"ID":"5113eabc-e5b6-4685-8d3f-e860e31befdd","Type":"ContainerStarted","Data":"825508ef9ba576ea4eb20fbf162b8e83500b22f7e6e675ccb54b25a92f56b302"} Jul 23 00:45:00.770890 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:00.770843 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" podStartSLOduration=0.770828376 podStartE2EDuration="770.828376ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:45:00.768766342 +0000 UTC m=+1530.065461312" watchObservedRunningTime="2026-07-23 00:45:00.770828376 +0000 UTC m=+1530.067523347" Jul 23 00:45:01.762031 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:01.762001 2585 generic.go:356] "Generic (PLEG): container finished" podID="5113eabc-e5b6-4685-8d3f-e860e31befdd" containerID="faae67a06c9ba1a959830e6e8870ac1ae67722f2ee6add28d25d3628f8821846" exitCode=0 Jul 23 00:45:01.762423 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:01.762085 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" event={"ID":"5113eabc-e5b6-4685-8d3f-e860e31befdd","Type":"ContainerDied","Data":"faae67a06c9ba1a959830e6e8870ac1ae67722f2ee6add28d25d3628f8821846"} Jul 23 00:45:02.908404 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:02.908375 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" Jul 23 00:45:03.779565 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:03.779523 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" event={"ID":"5113eabc-e5b6-4685-8d3f-e860e31befdd","Type":"ContainerDied","Data":"825508ef9ba576ea4eb20fbf162b8e83500b22f7e6e675ccb54b25a92f56b302"} Jul 23 00:45:03.779733 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:03.779527 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-4c4gp" Jul 23 00:45:03.779733 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:03.779608 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="825508ef9ba576ea4eb20fbf162b8e83500b22f7e6e675ccb54b25a92f56b302" Jul 23 00:45:47.358128 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:45:47.358090 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:05.143244 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:05.143213 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:08.320372 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:08.320334 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:08.420282 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:08.420248 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:08.815342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:08.815310 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:14.408546 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.408491 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:14.472072 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.472040 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:14.644348 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.644319 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:14.929926 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.929898 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc"] Jul 23 00:46:14.930333 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.930318 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5113eabc-e5b6-4685-8d3f-e860e31befdd" containerName="cleanup" Jul 23 00:46:14.930333 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.930335 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5113eabc-e5b6-4685-8d3f-e860e31befdd" containerName="cleanup" Jul 23 00:46:14.930460 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.930405 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5113eabc-e5b6-4685-8d3f-e860e31befdd" containerName="cleanup" Jul 23 00:46:14.933645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.933626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.935765 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.935742 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0b6bf1ce-openshift-default-dockercfg-2k264\"" Jul 23 00:46:14.943442 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.943422 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc"] Jul 23 00:46:14.951957 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.951934 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:14.962057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962036 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-token\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-credential-socket\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwz9\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-kube-api-access-4zwz9\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962148 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef0c718-3833-4d33-9572-2edf08d5fb07-istiod-ca-cert\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962299 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962203 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-envoy\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962299 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-podinfo\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962404 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-socket\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962404 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-data\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:14.962404 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:14.962376 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-certs\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063737 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef0c718-3833-4d33-9572-2edf08d5fb07-istiod-ca-cert\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-envoy\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-podinfo\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-socket\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-data\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-certs\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.063928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.063877 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-token\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064259 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064233 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-socket\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064312 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064282 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-data\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064391 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-credential-socket\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064452 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064436 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwz9\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-kube-api-access-4zwz9\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064452 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064446 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-certs\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064689 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064668 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef0c718-3833-4d33-9572-2edf08d5fb07-istiod-ca-cert\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.064761 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.064688 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-credential-socket\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.066120 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.066075 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-envoy\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.066495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.066456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-podinfo\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.077983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.077958 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-token\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.078768 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.078745 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwz9\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-kube-api-access-4zwz9\") pod \"e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.246379 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.246352 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:15.390732 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.390701 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc"] Jul 23 00:46:15.391610 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:46:15.391586 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef0c718_3833_4d33_9572_2edf08d5fb07.slice/crio-ce425d632eb31aa6a3a0c8d4c24047a3d2884c1effdc37249f5af7c1c4df8e20 WatchSource:0}: Error finding container ce425d632eb31aa6a3a0c8d4c24047a3d2884c1effdc37249f5af7c1c4df8e20: Status 404 returned error can't find the container with id ce425d632eb31aa6a3a0c8d4c24047a3d2884c1effdc37249f5af7c1c4df8e20 Jul 23 00:46:15.393825 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.393794 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:15.393904 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.393857 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:15.393904 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.393884 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:15.902244 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:15.902203 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:16.163285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:16.163194 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" event={"ID":"cef0c718-3833-4d33-9572-2edf08d5fb07","Type":"ContainerStarted","Data":"cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24"} Jul 23 00:46:16.163285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:16.163250 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" event={"ID":"cef0c718-3833-4d33-9572-2edf08d5fb07","Type":"ContainerStarted","Data":"ce425d632eb31aa6a3a0c8d4c24047a3d2884c1effdc37249f5af7c1c4df8e20"} Jul 23 00:46:16.180823 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:16.180772 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" podStartSLOduration=2.180757741 podStartE2EDuration="2.180757741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:46:16.179212423 +0000 UTC m=+1605.475907419" watchObservedRunningTime="2026-07-23 00:46:16.180757741 +0000 UTC m=+1605.477452709" Jul 23 00:46:16.246813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:16.246790 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:16.252169 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:16.252142 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:17.168074 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:17.168042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:17.169138 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:17.169119 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:17.515626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:17.515596 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:17.609549 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:17.609508 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:17.677630 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:17.677601 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:18.279947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:18.279905 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:18.383625 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:18.383598 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:18.512746 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:18.512716 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:24.201707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:24.201628 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:24.302133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:24.302100 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:32.325309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:32.325280 2585 patch_prober.go:28] interesting pod/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:47946->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 23 00:46:32.325794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:32.325355 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:47946->10.132.0.53:15021: read: connection reset by peer" Jul 23 00:46:33.263532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:33.263499 2585 generic.go:356] "Generic (PLEG): container finished" podID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerID="cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24" exitCode=0 Jul 23 00:46:33.263721 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:33.263581 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" event={"ID":"cef0c718-3833-4d33-9572-2edf08d5fb07","Type":"ContainerDied","Data":"cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24"} Jul 23 00:46:33.264024 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:33.264005 2585 scope.go:117] "RemoveContainer" containerID="cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24" Jul 23 00:46:33.265036 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:33.265007 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:33.265136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:33.265088 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:33.265201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:33.265153 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:34.270264 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:34.270225 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" event={"ID":"cef0c718-3833-4d33-9572-2edf08d5fb07","Type":"ContainerStarted","Data":"8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5"} Jul 23 00:46:35.247547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:35.247507 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:35.247845 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:35.247820 2585 patch_prober.go:28] interesting pod/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 00:46:35.247943 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:35.247888 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 00:46:36.247597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:36.247562 2585 patch_prober.go:28] interesting pod/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 00:46:36.248011 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:36.247618 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 00:46:37.247434 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:37.247390 2585 patch_prober.go:28] interesting pod/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 00:46:37.247644 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:37.247481 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 00:46:38.250694 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:38.250661 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:38.251074 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:38.251025 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:38.251787 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:38.251767 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:46:43.264849 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.264815 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj"] Jul 23 00:46:43.267271 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.267255 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:43.269134 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.269113 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:46:43.269415 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.269402 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:46:43.274833 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.274524 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj"] Jul 23 00:46:43.421393 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.421366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086a39e9-eaea-4f3f-8417-7082ef5235e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj\" (UID: \"086a39e9-eaea-4f3f-8417-7082ef5235e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:43.521872 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.521794 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086a39e9-eaea-4f3f-8417-7082ef5235e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj\" (UID: \"086a39e9-eaea-4f3f-8417-7082ef5235e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:43.522495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.522440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086a39e9-eaea-4f3f-8417-7082ef5235e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj\" (UID: \"086a39e9-eaea-4f3f-8417-7082ef5235e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:43.579080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.579054 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:43.712795 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:43.712767 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj"] Jul 23 00:46:43.714235 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:46:43.714202 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086a39e9_eaea_4f3f_8417_7082ef5235e1.slice/crio-e911d80176e4a674f2533b987a917c9dab3662f2a75b7356c5cfb2a9293fcc9b WatchSource:0}: Error finding container e911d80176e4a674f2533b987a917c9dab3662f2a75b7356c5cfb2a9293fcc9b: Status 404 returned error can't find the container with id e911d80176e4a674f2533b987a917c9dab3662f2a75b7356c5cfb2a9293fcc9b Jul 23 00:46:44.324173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:44.324140 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" event={"ID":"086a39e9-eaea-4f3f-8417-7082ef5235e1","Type":"ContainerStarted","Data":"d189e32853856965adaf6497079ca653a95424b590db89bede0228dffcb81ca8"} Jul 23 00:46:44.324668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:44.324193 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" event={"ID":"086a39e9-eaea-4f3f-8417-7082ef5235e1","Type":"ContainerStarted","Data":"e911d80176e4a674f2533b987a917c9dab3662f2a75b7356c5cfb2a9293fcc9b"} Jul 23 00:46:44.340249 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:44.340201 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" podStartSLOduration=1.3401856429999999 podStartE2EDuration="1.340185643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:46:44.337414015 +0000 UTC m=+1633.634108984" watchObservedRunningTime="2026-07-23 00:46:44.340185643 +0000 UTC m=+1633.636880611" Jul 23 00:46:45.331974 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:45.331935 2585 generic.go:356] "Generic (PLEG): container finished" podID="086a39e9-eaea-4f3f-8417-7082ef5235e1" containerID="d189e32853856965adaf6497079ca653a95424b590db89bede0228dffcb81ca8" exitCode=0 Jul 23 00:46:45.332336 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:45.332016 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" event={"ID":"086a39e9-eaea-4f3f-8417-7082ef5235e1","Type":"ContainerDied","Data":"d189e32853856965adaf6497079ca653a95424b590db89bede0228dffcb81ca8"} Jul 23 00:46:46.478118 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:46.478090 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:46.550834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:46.550805 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086a39e9-eaea-4f3f-8417-7082ef5235e1-maas-api-service-ca\") pod \"086a39e9-eaea-4f3f-8417-7082ef5235e1\" (UID: \"086a39e9-eaea-4f3f-8417-7082ef5235e1\") " Jul 23 00:46:46.551164 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:46.551140 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086a39e9-eaea-4f3f-8417-7082ef5235e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "086a39e9-eaea-4f3f-8417-7082ef5235e1" (UID: "086a39e9-eaea-4f3f-8417-7082ef5235e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:46:46.652448 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:46.652356 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086a39e9-eaea-4f3f-8417-7082ef5235e1-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:46:47.344213 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:47.344183 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" Jul 23 00:46:47.344362 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:47.344211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj" event={"ID":"086a39e9-eaea-4f3f-8417-7082ef5235e1","Type":"ContainerDied","Data":"e911d80176e4a674f2533b987a917c9dab3662f2a75b7356c5cfb2a9293fcc9b"} Jul 23 00:46:47.344362 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:47.344258 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e911d80176e4a674f2533b987a917c9dab3662f2a75b7356c5cfb2a9293fcc9b" Jul 23 00:46:53.691013 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:53.690979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:58.329514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:58.329475 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj"] Jul 23 00:46:58.331338 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:58.331314 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b6bf1ce-4qkvj"] Jul 23 00:46:58.334301 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:58.334273 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:58.470408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:58.470372 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc"] Jul 23 00:46:58.470641 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:58.470606 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" containerID="cri-o://8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5" gracePeriod=30 Jul 23 00:46:59.367998 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.367960 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:59.411331 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.411293 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="086a39e9-eaea-4f3f-8417-7082ef5235e1" path="/var/lib/kubelet/pods/086a39e9-eaea-4f3f-8417-7082ef5235e1/volumes" Jul 23 00:46:59.570568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.570537 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn"] Jul 23 00:46:59.570963 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.570950 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086a39e9-eaea-4f3f-8417-7082ef5235e1" containerName="revoke-keys" Jul 23 00:46:59.571008 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.570965 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="086a39e9-eaea-4f3f-8417-7082ef5235e1" containerName="revoke-keys" Jul 23 00:46:59.571041 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.571030 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="086a39e9-eaea-4f3f-8417-7082ef5235e1" containerName="revoke-keys" Jul 23 00:46:59.573123 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.573101 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.575380 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.575359 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-17c44e03-gw-openshift-default-dockercfg-t4wsg\"" Jul 23 00:46:59.586965 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.586945 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn"] Jul 23 00:46:59.600876 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.600849 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:59.664212 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-certs\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664212 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-data\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664239 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-token\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664280 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-envoy\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-socket\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-credential-socket\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664454 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istiod-ca-cert\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664499 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpkj\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-kube-api-access-8qpkj\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.664635 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.664534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-podinfo\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.765739 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765702 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-token\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.765935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-envoy\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.765935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765868 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-socket\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.765935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765898 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-credential-socket\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.765935 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istiod-ca-cert\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpkj\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-kube-api-access-8qpkj\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.765979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-podinfo\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-certs\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-data\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766360 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766304 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-socket\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766360 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766327 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-credential-socket\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766518 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766496 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-data\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766573 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766556 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-certs\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.766870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.766853 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istiod-ca-cert\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.768068 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.768045 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-envoy\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.768347 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.768329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-podinfo\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.776196 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.776173 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpkj\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-kube-api-access-8qpkj\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.777865 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.777843 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-token\") pod \"e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:46:59.809748 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.809725 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:59.883947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:46:59.883916 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:00.033355 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.033328 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn"] Jul 23 00:47:00.034974 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:47:00.034941 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9963f65_eb7d_44b4_a4ea_45e60365b7fa.slice/crio-d43d9abefefa945348fec59dda1a1fcd7433430fd822a2165c4d1cad0e8de935 WatchSource:0}: Error finding container d43d9abefefa945348fec59dda1a1fcd7433430fd822a2165c4d1cad0e8de935: Status 404 returned error can't find the container with id d43d9abefefa945348fec59dda1a1fcd7433430fd822a2165c4d1cad0e8de935 Jul 23 00:47:00.037035 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.037003 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:00.037151 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.037066 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:00.037151 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.037096 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:00.417865 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.417824 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" event={"ID":"f9963f65-eb7d-44b4-a4ea-45e60365b7fa","Type":"ContainerStarted","Data":"eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0"} Jul 23 00:47:00.418249 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.417884 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" event={"ID":"f9963f65-eb7d-44b4-a4ea-45e60365b7fa","Type":"ContainerStarted","Data":"d43d9abefefa945348fec59dda1a1fcd7433430fd822a2165c4d1cad0e8de935"} Jul 23 00:47:00.437120 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.437065 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podStartSLOduration=1.4370511910000001 podStartE2EDuration="1.437051191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:47:00.435345144 +0000 UTC m=+1649.732040114" watchObservedRunningTime="2026-07-23 00:47:00.437051191 +0000 UTC m=+1649.733746401" Jul 23 00:47:00.884179 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:00.884145 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:01.889392 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:01.889356 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:01.925160 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:01.925129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:02.089966 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:02.089937 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:02.428284 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:02.428254 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:02.429237 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:02.429214 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:02.455684 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:02.455658 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:02.556670 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:02.556638 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:02.658284 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:02.658256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:03.747107 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.747087 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:47:03.804321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-data\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804332 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-certs\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804357 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-podinfo\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804423 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-credential-socket\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804445 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-envoy\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804724 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804557 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef0c718-3833-4d33-9572-2edf08d5fb07-istiod-ca-cert\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804724 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804579 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-socket\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804724 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804612 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zwz9\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-kube-api-access-4zwz9\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.804724 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.804650 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-token\") pod \"cef0c718-3833-4d33-9572-2edf08d5fb07\" (UID: \"cef0c718-3833-4d33-9572-2edf08d5fb07\") " Jul 23 00:47:03.805411 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.805377 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:03.805565 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.805444 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:03.805565 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.805438 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef0c718-3833-4d33-9572-2edf08d5fb07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:47:03.807514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.806122 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:03.807514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.806282 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.807514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.806304 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef0c718-3833-4d33-9572-2edf08d5fb07-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.807514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.806318 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.807514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.806339 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.807514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.807122 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-data" (OuterVolumeSpecName: "istio-data") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:03.818560 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.818332 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:03.818560 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.818371 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:47:03.818741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.818646 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-kube-api-access-4zwz9" (OuterVolumeSpecName: "kube-api-access-4zwz9") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "kube-api-access-4zwz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:47:03.818741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.818712 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-token" (OuterVolumeSpecName: "istio-token") pod "cef0c718-3833-4d33-9572-2edf08d5fb07" (UID: "cef0c718-3833-4d33-9572-2edf08d5fb07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:47:03.907132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.907101 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zwz9\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-kube-api-access-4zwz9\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.907132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.907129 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.907132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.907139 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.907353 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.907146 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:03.907353 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:03.907154 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef0c718-3833-4d33-9572-2edf08d5fb07-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:04.439527 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.439457 2585 generic.go:356] "Generic (PLEG): container finished" podID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerID="8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5" exitCode=0 Jul 23 00:47:04.439712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.439531 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" event={"ID":"cef0c718-3833-4d33-9572-2edf08d5fb07","Type":"ContainerDied","Data":"8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5"} Jul 23 00:47:04.439712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.439548 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" Jul 23 00:47:04.439712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.439581 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc" event={"ID":"cef0c718-3833-4d33-9572-2edf08d5fb07","Type":"ContainerDied","Data":"ce425d632eb31aa6a3a0c8d4c24047a3d2884c1effdc37249f5af7c1c4df8e20"} Jul 23 00:47:04.439712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.439604 2585 scope.go:117] "RemoveContainer" containerID="8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5" Jul 23 00:47:04.448907 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.448889 2585 scope.go:117] "RemoveContainer" containerID="cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24" Jul 23 00:47:04.457067 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.457051 2585 scope.go:117] "RemoveContainer" containerID="8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5" Jul 23 00:47:04.457324 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:47:04.457303 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5\": container with ID starting with 8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5 not found: ID does not exist" containerID="8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5" Jul 23 00:47:04.457372 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.457332 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5"} err="failed to get container status \"8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5\": rpc error: code = NotFound desc = could not find container \"8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5\": container with ID starting with 8bdd6622efa5d105e5a7cf83917c89d9ef242a670894918c446a182d80eb75f5 not found: ID does not exist" Jul 23 00:47:04.457372 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.457348 2585 scope.go:117] "RemoveContainer" containerID="cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24" Jul 23 00:47:04.457592 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:47:04.457572 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24\": container with ID starting with cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24 not found: ID does not exist" containerID="cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24" Jul 23 00:47:04.457654 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.457602 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24"} err="failed to get container status \"cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24\": rpc error: code = NotFound desc = could not find container \"cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24\": container with ID starting with cc1cf4d9ad57fc3b8ed010a5f165174eef468807e84e4990f3e49fbd6b75aa24 not found: ID does not exist" Jul 23 00:47:04.460600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.460579 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc"] Jul 23 00:47:04.465577 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:04.465556 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0b6bf1ce-openshift-default-bf4ffc685-stwcc"] Jul 23 00:47:05.412757 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:05.412724 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" path="/var/lib/kubelet/pods/cef0c718-3833-4d33-9572-2edf08d5fb07/volumes" Jul 23 00:47:16.505749 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:16.505671 2585 generic.go:356] "Generic (PLEG): container finished" podID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerID="eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0" exitCode=0 Jul 23 00:47:16.505749 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:16.505731 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" event={"ID":"f9963f65-eb7d-44b4-a4ea-45e60365b7fa","Type":"ContainerDied","Data":"eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0"} Jul 23 00:47:16.506261 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:16.506212 2585 scope.go:117] "RemoveContainer" containerID="eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0" Jul 23 00:47:16.507282 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:16.507252 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:16.507388 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:16.507336 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:16.507449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:16.507408 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:17.428586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:17.428552 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:17.512741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:17.512708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" event={"ID":"f9963f65-eb7d-44b4-a4ea-45e60365b7fa","Type":"ContainerStarted","Data":"7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131"} Jul 23 00:47:17.884206 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:17.884168 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:17.885403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:17.885376 2585 patch_prober.go:28] interesting pod/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 00:47:17.885547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:17.885429 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 00:47:18.884374 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:18.884335 2585 patch_prober.go:28] interesting pod/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 00:47:18.884875 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:18.884392 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 00:47:19.885199 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:19.885162 2585 patch_prober.go:28] interesting pod/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 00:47:19.885628 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:19.885217 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 00:47:20.885206 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:20.885170 2585 patch_prober.go:28] interesting pod/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 00:47:20.885596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:20.885222 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 00:47:21.885305 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:21.885268 2585 patch_prober.go:28] interesting pod/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 00:47:21.885705 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:21.885337 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 00:47:22.884907 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:22.884870 2585 patch_prober.go:28] interesting pod/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 00:47:22.885079 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:22.884936 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 00:47:23.888101 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:23.888066 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:23.888822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:23.888794 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:23.896396 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:23.896370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:27.047453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.047424 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z"] Jul 23 00:47:27.047817 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.047809 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" Jul 23 00:47:27.047864 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.047821 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" Jul 23 00:47:27.047899 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.047892 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" Jul 23 00:47:27.047932 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.047903 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" Jul 23 00:47:27.050021 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.050005 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:27.053419 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.053399 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:47:27.053558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.053500 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:47:27.064679 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.064660 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z"] Jul 23 00:47:27.113648 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.113618 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e24480f-4f59-4206-920e-99aad01f2978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z\" (UID: \"1e24480f-4f59-4206-920e-99aad01f2978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:27.214645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.214612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e24480f-4f59-4206-920e-99aad01f2978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z\" (UID: \"1e24480f-4f59-4206-920e-99aad01f2978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:27.215221 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.215200 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e24480f-4f59-4206-920e-99aad01f2978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z\" (UID: \"1e24480f-4f59-4206-920e-99aad01f2978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:27.360426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.360341 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:27.494377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.494350 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z"] Jul 23 00:47:27.495936 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:47:27.495906 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e24480f_4f59_4206_920e_99aad01f2978.slice/crio-846c2041ab629b658e86056c1502b21590b7c2ed83cbb8a526c877a53dc485c7 WatchSource:0}: Error finding container 846c2041ab629b658e86056c1502b21590b7c2ed83cbb8a526c877a53dc485c7: Status 404 returned error can't find the container with id 846c2041ab629b658e86056c1502b21590b7c2ed83cbb8a526c877a53dc485c7 Jul 23 00:47:27.567425 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:27.567398 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" event={"ID":"1e24480f-4f59-4206-920e-99aad01f2978","Type":"ContainerStarted","Data":"846c2041ab629b658e86056c1502b21590b7c2ed83cbb8a526c877a53dc485c7"} Jul 23 00:47:28.574013 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:28.573977 2585 generic.go:356] "Generic (PLEG): container finished" podID="1e24480f-4f59-4206-920e-99aad01f2978" containerID="b91ca61b96fac1398c9162e4dcc0d194f17ef75212993fbdaf94045e01f74c04" exitCode=0 Jul 23 00:47:28.574381 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:28.574035 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" event={"ID":"1e24480f-4f59-4206-920e-99aad01f2978","Type":"ContainerDied","Data":"b91ca61b96fac1398c9162e4dcc0d194f17ef75212993fbdaf94045e01f74c04"} Jul 23 00:47:29.717506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:29.717480 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:29.835845 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:29.835764 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e24480f-4f59-4206-920e-99aad01f2978-maas-api-service-ca\") pod \"1e24480f-4f59-4206-920e-99aad01f2978\" (UID: \"1e24480f-4f59-4206-920e-99aad01f2978\") " Jul 23 00:47:29.836157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:29.836135 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e24480f-4f59-4206-920e-99aad01f2978-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e24480f-4f59-4206-920e-99aad01f2978" (UID: "1e24480f-4f59-4206-920e-99aad01f2978"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:47:29.936735 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:29.936703 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e24480f-4f59-4206-920e-99aad01f2978-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:30.586070 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:30.586029 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" event={"ID":"1e24480f-4f59-4206-920e-99aad01f2978","Type":"ContainerDied","Data":"846c2041ab629b658e86056c1502b21590b7c2ed83cbb8a526c877a53dc485c7"} Jul 23 00:47:30.586253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:30.586084 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846c2041ab629b658e86056c1502b21590b7c2ed83cbb8a526c877a53dc485c7" Jul 23 00:47:30.586253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:30.586052 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z" Jul 23 00:47:37.439384 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:37.439333 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:42.098996 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.098961 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z"] Jul 23 00:47:42.102043 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.102015 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:42.102285 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.102266 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17c44e03-pgm7z"] Jul 23 00:47:42.245312 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.245279 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn"] Jul 23 00:47:42.245618 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.245563 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" containerID="cri-o://7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131" gracePeriod=30 Jul 23 00:47:42.270844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.270795 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:42.519002 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.518969 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:42.819492 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:42.819390 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:43.092056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.091978 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5"] Jul 23 00:47:43.092531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.092513 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" Jul 23 00:47:43.092531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.092533 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef0c718-3833-4d33-9572-2edf08d5fb07" containerName="istio-proxy" Jul 23 00:47:43.092687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.092544 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e24480f-4f59-4206-920e-99aad01f2978" containerName="revoke-keys" Jul 23 00:47:43.092687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.092552 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e24480f-4f59-4206-920e-99aad01f2978" containerName="revoke-keys" Jul 23 00:47:43.092687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.092645 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e24480f-4f59-4206-920e-99aad01f2978" containerName="revoke-keys" Jul 23 00:47:43.095165 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.095143 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.097136 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.097117 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1648ae5-openshift-default-dockercfg-86c84\"" Jul 23 00:47:43.108827 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.108798 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5"] Jul 23 00:47:43.117890 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.117863 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:43.250450 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250414 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498d39c2-e0c8-45c9-8625-18e2900f7295-istiod-ca-cert\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-socket\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250517 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-podinfo\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-certs\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250608 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-token\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250767 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250651 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-credential-socket\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250767 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-data\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250767 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250699 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-envoy\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.250767 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.250754 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4f5n\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-kube-api-access-d4f5n\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-podinfo\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351636 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-certs\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351681 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-token\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-credential-socket\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351714 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-data\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-envoy\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351801 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f5n\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-kube-api-access-d4f5n\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498d39c2-e0c8-45c9-8625-18e2900f7295-istiod-ca-cert\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.351953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.351875 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-socket\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.352257 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.352058 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-certs\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.352257 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.352101 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-credential-socket\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.352257 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.352148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-data\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.352257 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.352179 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-socket\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.352617 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.352599 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498d39c2-e0c8-45c9-8625-18e2900f7295-istiod-ca-cert\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.354227 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.354203 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-envoy\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.354399 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.354384 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-podinfo\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.358901 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.358879 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-token\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.359093 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.359073 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4f5n\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-kube-api-access-d4f5n\") pod \"e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.412685 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.412653 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e24480f-4f59-4206-920e-99aad01f2978" path="/var/lib/kubelet/pods/1e24480f-4f59-4206-920e-99aad01f2978/volumes" Jul 23 00:47:43.413066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.413052 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:43.427021 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.426993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:43.558131 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.558105 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5"] Jul 23 00:47:43.560931 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:47:43.560901 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498d39c2_e0c8_45c9_8625_18e2900f7295.slice/crio-113d88356ca1db08f4a467935e115d92a0cbad8898310879063329a11c7ee6d9 WatchSource:0}: Error finding container 113d88356ca1db08f4a467935e115d92a0cbad8898310879063329a11c7ee6d9: Status 404 returned error can't find the container with id 113d88356ca1db08f4a467935e115d92a0cbad8898310879063329a11c7ee6d9 Jul 23 00:47:43.563071 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.563041 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:43.563183 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.563107 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:43.563183 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.563142 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:47:43.661223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.661186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" event={"ID":"498d39c2-e0c8-45c9-8625-18e2900f7295","Type":"ContainerStarted","Data":"2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c"} Jul 23 00:47:43.661373 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.661245 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" event={"ID":"498d39c2-e0c8-45c9-8625-18e2900f7295","Type":"ContainerStarted","Data":"113d88356ca1db08f4a467935e115d92a0cbad8898310879063329a11c7ee6d9"} Jul 23 00:47:43.680420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:43.680365 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podStartSLOduration=0.680351397 podStartE2EDuration="680.351397ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:47:43.678296584 +0000 UTC m=+1692.974991554" watchObservedRunningTime="2026-07-23 00:47:43.680351397 +0000 UTC m=+1692.977046419" Jul 23 00:47:44.413916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:44.413883 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:44.418901 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:44.418877 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:44.666908 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:44.666834 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:44.667726 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:44.667708 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:47:45.027224 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:45.027191 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:45.102671 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:45.102442 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:45.169876 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:45.169846 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:45.270262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:45.270233 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:45.697730 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:45.697694 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:45.798661 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:45.798623 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:47.505059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.505032 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:47.685282 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.685201 2585 generic.go:356] "Generic (PLEG): container finished" podID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerID="7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131" exitCode=0 Jul 23 00:47:47.685282 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.685251 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" Jul 23 00:47:47.685282 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.685271 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" event={"ID":"f9963f65-eb7d-44b4-a4ea-45e60365b7fa","Type":"ContainerDied","Data":"7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131"} Jul 23 00:47:47.685532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.685313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn" event={"ID":"f9963f65-eb7d-44b4-a4ea-45e60365b7fa","Type":"ContainerDied","Data":"d43d9abefefa945348fec59dda1a1fcd7433430fd822a2165c4d1cad0e8de935"} Jul 23 00:47:47.685532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.685334 2585 scope.go:117] "RemoveContainer" containerID="7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131" Jul 23 00:47:47.692298 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692278 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-socket\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692416 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692313 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-certs\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692416 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692329 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-envoy\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692416 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692345 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-credential-socket\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692416 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692382 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-token\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692414 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istiod-ca-cert\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692478 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qpkj\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-kube-api-access-8qpkj\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692507 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-podinfo\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692572 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:47.692653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692602 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-data\") pod \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\" (UID: \"f9963f65-eb7d-44b4-a4ea-45e60365b7fa\") " Jul 23 00:47:47.692653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692645 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:47.692976 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692945 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.692976 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.692965 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.693078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.693036 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:47.693288 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.693266 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:47:47.694172 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.694142 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:47.694771 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.694738 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-kube-api-access-8qpkj" (OuterVolumeSpecName: "kube-api-access-8qpkj") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "kube-api-access-8qpkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:47:47.694971 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.694953 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:47.695055 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.694972 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:47:47.695123 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.695108 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9963f65-eb7d-44b4-a4ea-45e60365b7fa" (UID: "f9963f65-eb7d-44b4-a4ea-45e60365b7fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:47:47.695430 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.695417 2585 scope.go:117] "RemoveContainer" containerID="eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0" Jul 23 00:47:47.714215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.714195 2585 scope.go:117] "RemoveContainer" containerID="7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131" Jul 23 00:47:47.714502 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:47:47.714456 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131\": container with ID starting with 7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131 not found: ID does not exist" containerID="7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131" Jul 23 00:47:47.714550 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.714512 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131"} err="failed to get container status \"7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131\": rpc error: code = NotFound desc = could not find container \"7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131\": container with ID starting with 7cefad1739dfa969ac80a17ff7a7618643e3edb04af63605507d74624f787131 not found: ID does not exist" Jul 23 00:47:47.714550 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.714529 2585 scope.go:117] "RemoveContainer" containerID="eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0" Jul 23 00:47:47.714772 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:47:47.714755 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0\": container with ID starting with eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0 not found: ID does not exist" containerID="eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0" Jul 23 00:47:47.714813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.714778 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0"} err="failed to get container status \"eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0\": rpc error: code = NotFound desc = could not find container \"eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0\": container with ID starting with eb8553a9a50ff7292a91f28930f4dfc0d985cf21f5df5d62db84a0f96bb324b0 not found: ID does not exist" Jul 23 00:47:47.793913 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793876 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.793913 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793903 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.793913 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793912 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.793913 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793921 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.794169 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793932 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.794169 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793940 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpkj\" (UniqueName: \"kubernetes.io/projected/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-kube-api-access-8qpkj\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:47.794169 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:47.793949 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9963f65-eb7d-44b4-a4ea-45e60365b7fa-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:47:48.006383 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:48.006356 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn"] Jul 23 00:47:48.010679 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:48.010654 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-17c44e03-gw-openshift-default-64fdf85b7f-8t5wn"] Jul 23 00:47:49.411949 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:49.411912 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" path="/var/lib/kubelet/pods/f9963f65-eb7d-44b4-a4ea-45e60365b7fa/volumes" Jul 23 00:47:59.759396 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:59.759294 2585 patch_prober.go:28] interesting pod/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:57058->10.132.0.57:15021: read: connection reset by peer" start-of-body= Jul 23 00:47:59.759396 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:47:59.759370 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:57058->10.132.0.57:15021: read: connection reset by peer" Jul 23 00:48:00.767596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:00.767565 2585 generic.go:356] "Generic (PLEG): container finished" podID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerID="2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c" exitCode=0 Jul 23 00:48:00.768073 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:00.767633 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" event={"ID":"498d39c2-e0c8-45c9-8625-18e2900f7295","Type":"ContainerDied","Data":"2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c"} Jul 23 00:48:00.768144 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:00.768103 2585 scope.go:117] "RemoveContainer" containerID="2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c" Jul 23 00:48:00.769075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:00.769047 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:00.769182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:00.769118 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:00.769182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:00.769150 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:01.775322 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:01.775292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" event={"ID":"498d39c2-e0c8-45c9-8625-18e2900f7295","Type":"ContainerStarted","Data":"8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7"} Jul 23 00:48:02.413676 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:02.413639 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:48:02.414019 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:02.413996 2585 patch_prober.go:28] interesting pod/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 00:48:02.414168 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:02.414144 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 00:48:03.415797 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:03.415758 2585 patch_prober.go:28] interesting pod/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 00:48:03.416265 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:03.415822 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 00:48:04.414536 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:04.414499 2585 patch_prober.go:28] interesting pod/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 00:48:04.414843 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:04.414801 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 00:48:05.413872 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:05.413829 2585 patch_prober.go:28] interesting pod/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 00:48:05.414376 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:05.413908 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 00:48:06.416637 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:06.416602 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:48:06.417078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:06.417058 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:48:06.417656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:06.417639 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:48:11.004491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.004443 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p"] Jul 23 00:48:11.005043 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.005023 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" Jul 23 00:48:11.005137 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.005046 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" Jul 23 00:48:11.005137 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.005056 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" Jul 23 00:48:11.005137 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.005067 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" Jul 23 00:48:11.005300 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.005192 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" Jul 23 00:48:11.013939 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.013910 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:11.015585 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.015556 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p"] Jul 23 00:48:11.016827 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.016398 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:48:11.016827 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.016661 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:48:11.095122 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.095090 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bc9fbc-d318-488f-8b72-d84c2ba2404c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p\" (UID: \"e6bc9fbc-d318-488f-8b72-d84c2ba2404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:11.195821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.195789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bc9fbc-d318-488f-8b72-d84c2ba2404c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p\" (UID: \"e6bc9fbc-d318-488f-8b72-d84c2ba2404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:11.196419 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.196400 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bc9fbc-d318-488f-8b72-d84c2ba2404c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p\" (UID: \"e6bc9fbc-d318-488f-8b72-d84c2ba2404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:11.326367 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.326296 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:11.461648 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.461622 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p"] Jul 23 00:48:11.463141 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:48:11.463108 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bc9fbc_d318_488f_8b72_d84c2ba2404c.slice/crio-187630be907a38e233b1390291b0963d3d5d2eb8d11515e3465e89e2fcd366e8 WatchSource:0}: Error finding container 187630be907a38e233b1390291b0963d3d5d2eb8d11515e3465e89e2fcd366e8: Status 404 returned error can't find the container with id 187630be907a38e233b1390291b0963d3d5d2eb8d11515e3465e89e2fcd366e8 Jul 23 00:48:11.829029 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.828997 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" event={"ID":"e6bc9fbc-d318-488f-8b72-d84c2ba2404c","Type":"ContainerStarted","Data":"0f8a1afc73f240cc82a0fe8c1c02ae3bca0a66b26477b10a00010c0047a56049"} Jul 23 00:48:11.829202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.829063 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" event={"ID":"e6bc9fbc-d318-488f-8b72-d84c2ba2404c","Type":"ContainerStarted","Data":"187630be907a38e233b1390291b0963d3d5d2eb8d11515e3465e89e2fcd366e8"} Jul 23 00:48:11.844586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:11.844545 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" podStartSLOduration=1.8445300690000002 podStartE2EDuration="1.844530069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:48:11.84268143 +0000 UTC m=+1721.139376417" watchObservedRunningTime="2026-07-23 00:48:11.844530069 +0000 UTC m=+1721.141225068" Jul 23 00:48:12.836981 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:12.836946 2585 generic.go:356] "Generic (PLEG): container finished" podID="e6bc9fbc-d318-488f-8b72-d84c2ba2404c" containerID="0f8a1afc73f240cc82a0fe8c1c02ae3bca0a66b26477b10a00010c0047a56049" exitCode=0 Jul 23 00:48:12.837418 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:12.837032 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" event={"ID":"e6bc9fbc-d318-488f-8b72-d84c2ba2404c","Type":"ContainerDied","Data":"0f8a1afc73f240cc82a0fe8c1c02ae3bca0a66b26477b10a00010c0047a56049"} Jul 23 00:48:13.978191 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:13.978168 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:14.018443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:14.018416 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bc9fbc-d318-488f-8b72-d84c2ba2404c-maas-api-service-ca\") pod \"e6bc9fbc-d318-488f-8b72-d84c2ba2404c\" (UID: \"e6bc9fbc-d318-488f-8b72-d84c2ba2404c\") " Jul 23 00:48:14.018787 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:14.018761 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6bc9fbc-d318-488f-8b72-d84c2ba2404c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6bc9fbc-d318-488f-8b72-d84c2ba2404c" (UID: "e6bc9fbc-d318-488f-8b72-d84c2ba2404c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:48:14.119425 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:14.119332 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bc9fbc-d318-488f-8b72-d84c2ba2404c-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:14.848657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:14.848632 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" Jul 23 00:48:14.848829 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:14.848676 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p" event={"ID":"e6bc9fbc-d318-488f-8b72-d84c2ba2404c","Type":"ContainerDied","Data":"187630be907a38e233b1390291b0963d3d5d2eb8d11515e3465e89e2fcd366e8"} Jul 23 00:48:14.848829 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:14.848717 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187630be907a38e233b1390291b0963d3d5d2eb8d11515e3465e89e2fcd366e8" Jul 23 00:48:21.402728 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:21.402697 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:26.126187 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:26.126154 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p"] Jul 23 00:48:26.128007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:26.127969 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:26.134723 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:26.134700 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1648ae5-ntv8p"] Jul 23 00:48:27.382941 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:27.382911 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5"] Jul 23 00:48:27.383408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:27.383106 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" containerID="cri-o://8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7" gracePeriod=30 Jul 23 00:48:27.403205 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:27.403181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:27.411599 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:27.411575 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6bc9fbc-d318-488f-8b72-d84c2ba2404c" path="/var/lib/kubelet/pods/e6bc9fbc-d318-488f-8b72-d84c2ba2404c/volumes" Jul 23 00:48:27.636532 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:27.636411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:27.877283 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:27.877247 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:28.167249 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.167220 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m"] Jul 23 00:48:28.167656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.167635 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6bc9fbc-d318-488f-8b72-d84c2ba2404c" containerName="revoke-keys" Jul 23 00:48:28.167656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.167652 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6bc9fbc-d318-488f-8b72-d84c2ba2404c" containerName="revoke-keys" Jul 23 00:48:28.167844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.167711 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6bc9fbc-d318-488f-8b72-d84c2ba2404c" containerName="revoke-keys" Jul 23 00:48:28.167844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.167719 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9963f65-eb7d-44b4-a4ea-45e60365b7fa" containerName="istio-proxy" Jul 23 00:48:28.170066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.170049 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.171949 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.171932 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-627b9dcc-openshift-default-dockercfg-jdvfc\"" Jul 23 00:48:28.181439 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.181409 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m"] Jul 23 00:48:28.197982 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.197780 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:28.253587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-socket\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.253777 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253597 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-data\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.253777 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253723 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-certs\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.253894 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-credential-socket\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.253894 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253833 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c801f80c-b6ba-402d-8647-be9d71e02329-istio-podinfo\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.253894 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c801f80c-b6ba-402d-8647-be9d71e02329-istiod-ca-cert\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.254084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-envoy\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.254084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbz76\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-kube-api-access-hbz76\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.254084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.253999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-istio-token\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354662 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-certs\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354690 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-credential-socket\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354714 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c801f80c-b6ba-402d-8647-be9d71e02329-istio-podinfo\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354737 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c801f80c-b6ba-402d-8647-be9d71e02329-istiod-ca-cert\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354787 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-envoy\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbz76\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-kube-api-access-hbz76\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.354856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-istio-token\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-socket\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.354925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-data\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.355096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-credential-socket\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.355154 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-certs\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355447 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.355428 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-data\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355527 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.355452 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-socket\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.355569 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.355544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c801f80c-b6ba-402d-8647-be9d71e02329-istiod-ca-cert\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.357383 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.357353 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-envoy\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.357713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.357693 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c801f80c-b6ba-402d-8647-be9d71e02329-istio-podinfo\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.362501 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.362449 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-istio-token\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.362911 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.362894 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbz76\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-kube-api-access-hbz76\") pod \"e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.434586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.434501 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:28.482658 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.482627 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:28.623589 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.623561 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m"] Jul 23 00:48:28.625511 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:48:28.625448 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc801f80c_b6ba_402d_8647_be9d71e02329.slice/crio-be98e9ffa62b19837f90a6871dabb1ae018729b8c99d8cd9208b77e78831cdfa WatchSource:0}: Error finding container be98e9ffa62b19837f90a6871dabb1ae018729b8c99d8cd9208b77e78831cdfa: Status 404 returned error can't find the container with id be98e9ffa62b19837f90a6871dabb1ae018729b8c99d8cd9208b77e78831cdfa Jul 23 00:48:28.627717 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.627681 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:28.627805 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.627749 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:28.627805 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.627778 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:28.923513 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.923480 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" event={"ID":"c801f80c-b6ba-402d-8647-be9d71e02329","Type":"ContainerStarted","Data":"565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58"} Jul 23 00:48:28.923684 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.923534 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" event={"ID":"c801f80c-b6ba-402d-8647-be9d71e02329","Type":"ContainerStarted","Data":"be98e9ffa62b19837f90a6871dabb1ae018729b8c99d8cd9208b77e78831cdfa"} Jul 23 00:48:28.948224 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:28.948114 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podStartSLOduration=0.948093527 podStartE2EDuration="948.093527ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:48:28.946447586 +0000 UTC m=+1738.243142545" watchObservedRunningTime="2026-07-23 00:48:28.948093527 +0000 UTC m=+1738.244788496" Jul 23 00:48:29.482909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:29.482875 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:29.487837 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:29.487814 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:29.929366 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:29.929290 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:29.930272 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:29.930255 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:30.276374 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:30.276336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:30.382128 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:30.382082 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:30.454400 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:30.454370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:30.565800 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:30.565672 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:31.017364 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:31.017313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:31.122275 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:31.122202 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:32.638417 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.638394 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:48:32.803885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.803836 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-podinfo\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.803885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.803884 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-data\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.803915 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-certs\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.803941 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-envoy\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.803972 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498d39c2-e0c8-45c9-8625-18e2900f7295-istiod-ca-cert\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.803995 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-credential-socket\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804368 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804329 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:32.804368 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804333 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:32.804502 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804438 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498d39c2-e0c8-45c9-8625-18e2900f7295-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:48:32.804568 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804517 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-token\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804628 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804579 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-socket\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804684 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804633 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4f5n\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-kube-api-access-d4f5n\") pod \"498d39c2-e0c8-45c9-8625-18e2900f7295\" (UID: \"498d39c2-e0c8-45c9-8625-18e2900f7295\") " Jul 23 00:48:32.804801 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.804773 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-data" (OuterVolumeSpecName: "istio-data") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:32.805060 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.805027 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:32.805369 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.805347 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.805491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.805374 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.805491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.805388 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498d39c2-e0c8-45c9-8625-18e2900f7295-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.805491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.805400 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.805491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.805415 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.806769 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.806741 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:48:32.806889 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.806821 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:32.807063 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.807040 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-token" (OuterVolumeSpecName: "istio-token") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:48:32.807194 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.807178 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-kube-api-access-d4f5n" (OuterVolumeSpecName: "kube-api-access-d4f5n") pod "498d39c2-e0c8-45c9-8625-18e2900f7295" (UID: "498d39c2-e0c8-45c9-8625-18e2900f7295"). InnerVolumeSpecName "kube-api-access-d4f5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:48:32.906909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.906860 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.906909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.906901 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.906909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.906912 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4f5n\" (UniqueName: \"kubernetes.io/projected/498d39c2-e0c8-45c9-8625-18e2900f7295-kube-api-access-d4f5n\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.906909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.906922 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498d39c2-e0c8-45c9-8625-18e2900f7295-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:32.946861 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.946823 2585 generic.go:356] "Generic (PLEG): container finished" podID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerID="8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7" exitCode=0 Jul 23 00:48:32.947028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.946895 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" Jul 23 00:48:32.947028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.946911 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" event={"ID":"498d39c2-e0c8-45c9-8625-18e2900f7295","Type":"ContainerDied","Data":"8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7"} Jul 23 00:48:32.947028 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.946998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5" event={"ID":"498d39c2-e0c8-45c9-8625-18e2900f7295","Type":"ContainerDied","Data":"113d88356ca1db08f4a467935e115d92a0cbad8898310879063329a11c7ee6d9"} Jul 23 00:48:32.947148 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.947029 2585 scope.go:117] "RemoveContainer" containerID="8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7" Jul 23 00:48:32.956836 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.956819 2585 scope.go:117] "RemoveContainer" containerID="2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c" Jul 23 00:48:32.965392 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.965369 2585 scope.go:117] "RemoveContainer" containerID="8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7" Jul 23 00:48:32.965673 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:48:32.965653 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7\": container with ID starting with 8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7 not found: ID does not exist" containerID="8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7" Jul 23 00:48:32.965779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.965683 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7"} err="failed to get container status \"8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7\": rpc error: code = NotFound desc = could not find container \"8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7\": container with ID starting with 8aacf8c114e31a2aaf43cc07f44331924fdd516031d7fa2ca47319dc4e1dc3d7 not found: ID does not exist" Jul 23 00:48:32.965779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.965702 2585 scope.go:117] "RemoveContainer" containerID="2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c" Jul 23 00:48:32.965990 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:48:32.965965 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c\": container with ID starting with 2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c not found: ID does not exist" containerID="2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c" Jul 23 00:48:32.966049 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.965994 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c"} err="failed to get container status \"2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c\": rpc error: code = NotFound desc = could not find container \"2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c\": container with ID starting with 2d66d2ab9ca7d380370003b7b27acf114047dcb53a3ad2ce112de96206005c7c not found: ID does not exist" Jul 23 00:48:32.967672 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.967652 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5"] Jul 23 00:48:32.971697 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:32.971674 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1648ae5-openshift-default-7b9c88d597-vm9k5"] Jul 23 00:48:33.413199 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:33.413164 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" path="/var/lib/kubelet/pods/498d39c2-e0c8-45c9-8625-18e2900f7295/volumes" Jul 23 00:48:41.540497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:41.540447 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:41.640641 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:41.640609 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:44.930398 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:44.930360 2585 patch_prober.go:28] interesting pod/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 23 00:48:44.930814 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:44.930413 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 23 00:48:45.020882 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:45.020845 2585 generic.go:356] "Generic (PLEG): container finished" podID="c801f80c-b6ba-402d-8647-be9d71e02329" containerID="565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58" exitCode=0 Jul 23 00:48:45.021095 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:45.020912 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" event={"ID":"c801f80c-b6ba-402d-8647-be9d71e02329","Type":"ContainerDied","Data":"565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58"} Jul 23 00:48:45.021384 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:45.021367 2585 scope.go:117] "RemoveContainer" containerID="565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58" Jul 23 00:48:45.022431 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:45.022406 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:45.022509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:45.022492 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:45.022556 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:45.022526 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:46.031869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:46.031838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" event={"ID":"c801f80c-b6ba-402d-8647-be9d71e02329","Type":"ContainerStarted","Data":"983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a"} Jul 23 00:48:46.483253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:46.483223 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:46.483497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:46.483448 2585 patch_prober.go:28] interesting pod/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 23 00:48:46.483550 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:46.483523 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 23 00:48:47.484006 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:47.483972 2585 patch_prober.go:28] interesting pod/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 23 00:48:47.484381 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:47.484028 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 23 00:48:48.483392 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:48.483361 2585 patch_prober.go:28] interesting pod/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 23 00:48:48.483584 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:48.483419 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 23 00:48:49.482964 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:49.482928 2585 patch_prober.go:28] interesting pod/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 23 00:48:49.483341 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:49.482979 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 23 00:48:50.487391 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:50.487353 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:50.487791 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:50.487671 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:50.488273 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:50.488256 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:48:51.743329 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:51.743299 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:51.845155 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:51.845100 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:55.250384 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.250347 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6"] Jul 23 00:48:55.250883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.250761 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" Jul 23 00:48:55.250883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.250775 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" Jul 23 00:48:55.250883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.250790 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" Jul 23 00:48:55.250883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.250795 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" Jul 23 00:48:55.250883 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.250861 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" Jul 23 00:48:55.252987 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.252971 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:55.254947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.254928 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:48:55.254947 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.254942 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:48:55.261159 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.261136 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6"] Jul 23 00:48:55.308763 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.308732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15f43fe-d876-4f48-9fcf-c004adbff14b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6\" (UID: \"b15f43fe-d876-4f48-9fcf-c004adbff14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:55.409224 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.409185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15f43fe-d876-4f48-9fcf-c004adbff14b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6\" (UID: \"b15f43fe-d876-4f48-9fcf-c004adbff14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:55.409863 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.409842 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15f43fe-d876-4f48-9fcf-c004adbff14b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6\" (UID: \"b15f43fe-d876-4f48-9fcf-c004adbff14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:55.564986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.564892 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:55.700879 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:55.700855 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6"] Jul 23 00:48:55.703033 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:48:55.702995 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15f43fe_d876_4f48_9fcf_c004adbff14b.slice/crio-60e04b5335a756fd0ddbac06119df7152bdedaedad97d7829b53cbce38286cf8 WatchSource:0}: Error finding container 60e04b5335a756fd0ddbac06119df7152bdedaedad97d7829b53cbce38286cf8: Status 404 returned error can't find the container with id 60e04b5335a756fd0ddbac06119df7152bdedaedad97d7829b53cbce38286cf8 Jul 23 00:48:56.088084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:56.088054 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" event={"ID":"b15f43fe-d876-4f48-9fcf-c004adbff14b","Type":"ContainerStarted","Data":"a921be5f63b8fdb813fb0782413ba500418481a55508e7d6ec0c59862a7b13c7"} Jul 23 00:48:56.088240 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:56.088105 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" event={"ID":"b15f43fe-d876-4f48-9fcf-c004adbff14b","Type":"ContainerStarted","Data":"60e04b5335a756fd0ddbac06119df7152bdedaedad97d7829b53cbce38286cf8"} Jul 23 00:48:56.107026 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:56.106983 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" podStartSLOduration=1.106968644 podStartE2EDuration="1.106968644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:48:56.104018193 +0000 UTC m=+1765.400713180" watchObservedRunningTime="2026-07-23 00:48:56.106968644 +0000 UTC m=+1765.403663612" Jul 23 00:48:57.094888 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:57.094850 2585 generic.go:356] "Generic (PLEG): container finished" podID="b15f43fe-d876-4f48-9fcf-c004adbff14b" containerID="a921be5f63b8fdb813fb0782413ba500418481a55508e7d6ec0c59862a7b13c7" exitCode=0 Jul 23 00:48:57.095262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:57.094937 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" event={"ID":"b15f43fe-d876-4f48-9fcf-c004adbff14b","Type":"ContainerDied","Data":"a921be5f63b8fdb813fb0782413ba500418481a55508e7d6ec0c59862a7b13c7"} Jul 23 00:48:58.241328 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:58.241304 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:58.334990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:58.334959 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15f43fe-d876-4f48-9fcf-c004adbff14b-maas-api-service-ca\") pod \"b15f43fe-d876-4f48-9fcf-c004adbff14b\" (UID: \"b15f43fe-d876-4f48-9fcf-c004adbff14b\") " Jul 23 00:48:58.335305 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:58.335282 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15f43fe-d876-4f48-9fcf-c004adbff14b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b15f43fe-d876-4f48-9fcf-c004adbff14b" (UID: "b15f43fe-d876-4f48-9fcf-c004adbff14b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:48:58.436451 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:58.436363 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15f43fe-d876-4f48-9fcf-c004adbff14b-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:48:59.107433 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:59.107406 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" Jul 23 00:48:59.107433 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:59.107417 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6" event={"ID":"b15f43fe-d876-4f48-9fcf-c004adbff14b","Type":"ContainerDied","Data":"60e04b5335a756fd0ddbac06119df7152bdedaedad97d7829b53cbce38286cf8"} Jul 23 00:48:59.107657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:48:59.107480 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e04b5335a756fd0ddbac06119df7152bdedaedad97d7829b53cbce38286cf8" Jul 23 00:49:05.676802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:05.676768 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.318130 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.318098 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.318130 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.318122 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6"] Jul 23 00:49:10.322146 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.322119 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-627b9dcc-z8dt6"] Jul 23 00:49:10.469449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.469413 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m"] Jul 23 00:49:10.469766 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.469735 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" containerID="cri-o://983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a" gracePeriod=30 Jul 23 00:49:10.488042 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.487980 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.816170 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:10.816119 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:11.324677 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:11.324646 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:11.411527 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:11.411494 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b15f43fe-d876-4f48-9fcf-c004adbff14b" path="/var/lib/kubelet/pods/b15f43fe-d876-4f48-9fcf-c004adbff14b/volumes" Jul 23 00:49:11.422980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:11.422956 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:11.521315 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:11.521285 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:13.031848 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:13.031807 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:15.723213 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.723190 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:49:15.783484 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783402 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-envoy\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783484 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783434 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-socket\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783484 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783452 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-credential-socket\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783493 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-istio-token\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783534 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-certs\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783567 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbz76\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-kube-api-access-hbz76\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783607 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-data\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783644 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c801f80c-b6ba-402d-8647-be9d71e02329-istiod-ca-cert\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783693 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c801f80c-b6ba-402d-8647-be9d71e02329-istio-podinfo\") pod \"c801f80c-b6ba-402d-8647-be9d71e02329\" (UID: \"c801f80c-b6ba-402d-8647-be9d71e02329\") " Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783720 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:49:15.783958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.783741 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:49:15.784376 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.784030 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.784376 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.784036 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c801f80c-b6ba-402d-8647-be9d71e02329-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:49:15.784376 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.784051 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.784376 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.784214 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:49:15.784677 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.784653 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-data" (OuterVolumeSpecName: "istio-data") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:49:15.785760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.785731 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:49:15.785860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.785835 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c801f80c-b6ba-402d-8647-be9d71e02329-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:49:15.786281 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.786254 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-istio-token" (OuterVolumeSpecName: "istio-token") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:49:15.786365 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.786280 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-kube-api-access-hbz76" (OuterVolumeSpecName: "kube-api-access-hbz76") pod "c801f80c-b6ba-402d-8647-be9d71e02329" (UID: "c801f80c-b6ba-402d-8647-be9d71e02329"). InnerVolumeSpecName "kube-api-access-hbz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:49:15.884961 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884930 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.884961 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884955 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.884961 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884964 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.885209 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884974 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbz76\" (UniqueName: \"kubernetes.io/projected/c801f80c-b6ba-402d-8647-be9d71e02329-kube-api-access-hbz76\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.885209 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884984 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c801f80c-b6ba-402d-8647-be9d71e02329-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.885209 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884992 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c801f80c-b6ba-402d-8647-be9d71e02329-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:15.885209 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:15.884999 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c801f80c-b6ba-402d-8647-be9d71e02329-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:49:16.200090 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.200009 2585 generic.go:356] "Generic (PLEG): container finished" podID="c801f80c-b6ba-402d-8647-be9d71e02329" containerID="983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a" exitCode=0 Jul 23 00:49:16.200090 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.200056 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" Jul 23 00:49:16.200090 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.200075 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" event={"ID":"c801f80c-b6ba-402d-8647-be9d71e02329","Type":"ContainerDied","Data":"983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a"} Jul 23 00:49:16.200337 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.200117 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m" event={"ID":"c801f80c-b6ba-402d-8647-be9d71e02329","Type":"ContainerDied","Data":"be98e9ffa62b19837f90a6871dabb1ae018729b8c99d8cd9208b77e78831cdfa"} Jul 23 00:49:16.200337 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.200139 2585 scope.go:117] "RemoveContainer" containerID="983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a" Jul 23 00:49:16.210996 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.210978 2585 scope.go:117] "RemoveContainer" containerID="565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58" Jul 23 00:49:16.219364 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.219343 2585 scope.go:117] "RemoveContainer" containerID="983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a" Jul 23 00:49:16.219631 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:49:16.219614 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a\": container with ID starting with 983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a not found: ID does not exist" containerID="983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a" Jul 23 00:49:16.219698 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.219639 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a"} err="failed to get container status \"983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a\": rpc error: code = NotFound desc = could not find container \"983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a\": container with ID starting with 983d367ce87d530018d82e0e749391527b62ebf9834530723d81755d0015d25a not found: ID does not exist" Jul 23 00:49:16.219698 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.219656 2585 scope.go:117] "RemoveContainer" containerID="565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58" Jul 23 00:49:16.219878 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:49:16.219862 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58\": container with ID starting with 565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58 not found: ID does not exist" containerID="565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58" Jul 23 00:49:16.219921 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.219882 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58"} err="failed to get container status \"565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58\": rpc error: code = NotFound desc = could not find container \"565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58\": container with ID starting with 565a147e27b74b3fe1a49bee8ac4d3d52274a2fc730d632ad1be91c6ee192d58 not found: ID does not exist" Jul 23 00:49:16.228164 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.228143 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m"] Jul 23 00:49:16.231356 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:16.231338 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-627b9dcc-openshift-default-7b7fdc8f67-k4n6m"] Jul 23 00:49:17.413392 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:17.413359 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" path="/var/lib/kubelet/pods/c801f80c-b6ba-402d-8647-be9d71e02329/volumes" Jul 23 00:49:19.371933 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:19.371898 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:19.468100 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:19.468064 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:19.569416 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:19.569378 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:31.279541 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:31.279512 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:49:31.282561 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:31.282540 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:49:45.576778 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:45.576746 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:51.933678 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:51.933632 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:52.035005 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:52.034974 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:52.135781 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:49:52.135749 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:08.073720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:08.073688 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:13.825453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:13.825416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:13.925238 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:13.925187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:14.025111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:14.025077 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:14.125222 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:14.125129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:24.629456 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:24.629416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:25.466839 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:25.466802 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:31.948818 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:31.948780 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:32.060479 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:32.060440 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:32.169235 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:32.169201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:32.268855 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:32.268821 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:33.082286 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:33.082253 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:33.180182 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:33.180149 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:33.279884 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:33.279857 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:35.003208 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:35.003170 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:40.823786 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:40.823742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:40.923517 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:40.923446 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:41.513952 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:41.513920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:47.367693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.367659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:47.424401 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.424367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:47.615884 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.615854 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:47.908441 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.908406 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9"] Jul 23 00:50:47.908996 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.908976 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b15f43fe-d876-4f48-9fcf-c004adbff14b" containerName="revoke-keys" Jul 23 00:50:47.909177 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909158 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b15f43fe-d876-4f48-9fcf-c004adbff14b" containerName="revoke-keys" Jul 23 00:50:47.909299 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909201 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" Jul 23 00:50:47.909299 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909209 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" Jul 23 00:50:47.909299 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909233 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" Jul 23 00:50:47.909299 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909241 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" Jul 23 00:50:47.909488 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909344 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b15f43fe-d876-4f48-9fcf-c004adbff14b" containerName="revoke-keys" Jul 23 00:50:47.909488 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909357 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" Jul 23 00:50:47.909488 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909368 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="498d39c2-e0c8-45c9-8625-18e2900f7295" containerName="istio-proxy" Jul 23 00:50:47.909488 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.909380 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c801f80c-b6ba-402d-8647-be9d71e02329" containerName="istio-proxy" Jul 23 00:50:47.913207 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.913187 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.916539 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.916521 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-89b85334-openshift-default-dockercfg-cvhmv\"" Jul 23 00:50:47.949566 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.949540 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9"] Jul 23 00:50:47.969597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-credential-socket\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969759 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969602 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-data\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969759 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ebe837-5275-464d-861d-cc6825b54ef5-istio-podinfo\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969747 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ebe837-5275-464d-861d-cc6825b54ef5-istiod-ca-cert\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-istio-token\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-certs\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-socket\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.969928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969917 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-envoy\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.970037 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.969942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6trz\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-kube-api-access-v6trz\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:47.980213 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:47.980185 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:48.071036 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071001 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ebe837-5275-464d-861d-cc6825b54ef5-istiod-ca-cert\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071210 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071047 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-istio-token\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071210 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-certs\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071331 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071234 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-socket\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071331 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071264 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-envoy\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071331 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071292 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6trz\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-kube-api-access-v6trz\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071535 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071399 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-credential-socket\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071535 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071422 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-data\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071535 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ebe837-5275-464d-861d-cc6825b54ef5-istio-podinfo\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-certs\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071636 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-socket\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071806 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071778 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ebe837-5275-464d-861d-cc6825b54ef5-istiod-ca-cert\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.071972 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.071927 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-data\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.072077 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.072033 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-credential-socket\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.073504 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.073485 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-envoy\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.073884 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.073863 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ebe837-5275-464d-861d-cc6825b54ef5-istio-podinfo\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.103399 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.097283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-istio-token\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.105422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.105392 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6trz\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-kube-api-access-v6trz\") pod \"e2e-mt-89b85334-openshift-default-d9dfb6595-875k9\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.223679 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.223576 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:48.378500 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.378459 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9"] Jul 23 00:50:48.380542 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:50:48.380514 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ebe837_5275_464d_861d_cc6825b54ef5.slice/crio-a111816f42573fc0ae1f72b07cc4a75ab2a0c87e4446b3470a08b6f64ba51ec4 WatchSource:0}: Error finding container a111816f42573fc0ae1f72b07cc4a75ab2a0c87e4446b3470a08b6f64ba51ec4: Status 404 returned error can't find the container with id a111816f42573fc0ae1f72b07cc4a75ab2a0c87e4446b3470a08b6f64ba51ec4 Jul 23 00:50:48.382649 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.382615 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:50:48.382729 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.382682 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:50:48.382729 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.382710 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:50:48.639754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.639723 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:48.694875 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.694843 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" event={"ID":"07ebe837-5275-464d-861d-cc6825b54ef5","Type":"ContainerStarted","Data":"6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785"} Jul 23 00:50:48.695084 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.694898 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" event={"ID":"07ebe837-5275-464d-861d-cc6825b54ef5","Type":"ContainerStarted","Data":"a111816f42573fc0ae1f72b07cc4a75ab2a0c87e4446b3470a08b6f64ba51ec4"} Jul 23 00:50:48.714978 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:48.714926 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podStartSLOduration=1.714910038 podStartE2EDuration="1.714910038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:50:48.712662264 +0000 UTC m=+1878.009357231" watchObservedRunningTime="2026-07-23 00:50:48.714910038 +0000 UTC m=+1878.011605005" Jul 23 00:50:49.224126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:49.224087 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:49.229713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:49.229686 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:49.700209 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:49.700176 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:49.701408 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:49.701391 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:50:50.869684 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:50.869649 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:50.952852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:50.952815 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:51.022031 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:51.021992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:51.214173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:51.214063 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:51.621926 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:51.621891 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:51.726917 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:51.726879 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:55.452454 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:55.452372 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:56.164723 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:56.164690 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:56.264910 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:56.264879 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:56.603118 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:50:56.603085 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:02.518701 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.518657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:02.619168 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.619123 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:02.919944 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.919860 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm"] Jul 23 00:51:02.924769 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.924744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:02.926984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.926961 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:51:02.926984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.926981 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:51:02.931860 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:02.931800 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm"] Jul 23 00:51:03.014223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.014185 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87362ca0-c02e-492d-9cee-896f45984347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-89b85334-vzfxm\" (UID: \"87362ca0-c02e-492d-9cee-896f45984347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:03.115570 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.115529 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87362ca0-c02e-492d-9cee-896f45984347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-89b85334-vzfxm\" (UID: \"87362ca0-c02e-492d-9cee-896f45984347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:03.116181 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.116155 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87362ca0-c02e-492d-9cee-896f45984347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-89b85334-vzfxm\" (UID: \"87362ca0-c02e-492d-9cee-896f45984347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:03.238058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.238024 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:03.380415 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.380390 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm"] Jul 23 00:51:03.381956 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:51:03.381929 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87362ca0_c02e_492d_9cee_896f45984347.slice/crio-39416b08e2f5ea4792658d8e0b6755bb5327b65092c56579928f38f68525677a WatchSource:0}: Error finding container 39416b08e2f5ea4792658d8e0b6755bb5327b65092c56579928f38f68525677a: Status 404 returned error can't find the container with id 39416b08e2f5ea4792658d8e0b6755bb5327b65092c56579928f38f68525677a Jul 23 00:51:03.778969 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.778936 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" event={"ID":"87362ca0-c02e-492d-9cee-896f45984347","Type":"ContainerStarted","Data":"e9d95e8ba62d609a50c1accb2c00dc189332f9fe8c07465dbb6ad8149ee891c4"} Jul 23 00:51:03.779347 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:03.778992 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" event={"ID":"87362ca0-c02e-492d-9cee-896f45984347","Type":"ContainerStarted","Data":"39416b08e2f5ea4792658d8e0b6755bb5327b65092c56579928f38f68525677a"} Jul 23 00:51:04.786321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:04.786287 2585 generic.go:356] "Generic (PLEG): container finished" podID="87362ca0-c02e-492d-9cee-896f45984347" containerID="e9d95e8ba62d609a50c1accb2c00dc189332f9fe8c07465dbb6ad8149ee891c4" exitCode=0 Jul 23 00:51:04.787691 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:04.786378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" event={"ID":"87362ca0-c02e-492d-9cee-896f45984347","Type":"ContainerDied","Data":"e9d95e8ba62d609a50c1accb2c00dc189332f9fe8c07465dbb6ad8149ee891c4"} Jul 23 00:51:04.804107 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:04.804079 2585 patch_prober.go:28] interesting pod/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:59128->10.132.0.61:15021: read: connection reset by peer" start-of-body= Jul 23 00:51:04.804244 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:04.804151 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:59128->10.132.0.61:15021: read: connection reset by peer" Jul 23 00:51:05.792711 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.792677 2585 generic.go:356] "Generic (PLEG): container finished" podID="07ebe837-5275-464d-861d-cc6825b54ef5" containerID="6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785" exitCode=0 Jul 23 00:51:05.793098 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.792749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" event={"ID":"07ebe837-5275-464d-861d-cc6825b54ef5","Type":"ContainerDied","Data":"6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785"} Jul 23 00:51:05.793492 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.793169 2585 scope.go:117] "RemoveContainer" containerID="6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785" Jul 23 00:51:05.794217 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.794171 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:05.794310 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.794252 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:05.794310 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.794282 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:05.947314 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:05.947289 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:06.040815 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.040787 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87362ca0-c02e-492d-9cee-896f45984347-maas-api-service-ca\") pod \"87362ca0-c02e-492d-9cee-896f45984347\" (UID: \"87362ca0-c02e-492d-9cee-896f45984347\") " Jul 23 00:51:06.041216 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.041190 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87362ca0-c02e-492d-9cee-896f45984347-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87362ca0-c02e-492d-9cee-896f45984347" (UID: "87362ca0-c02e-492d-9cee-896f45984347"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:51:06.141781 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.141705 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87362ca0-c02e-492d-9cee-896f45984347-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:06.799397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.799366 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" Jul 23 00:51:06.800135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.799383 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm" event={"ID":"87362ca0-c02e-492d-9cee-896f45984347","Type":"ContainerDied","Data":"39416b08e2f5ea4792658d8e0b6755bb5327b65092c56579928f38f68525677a"} Jul 23 00:51:06.800278 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.800172 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39416b08e2f5ea4792658d8e0b6755bb5327b65092c56579928f38f68525677a" Jul 23 00:51:06.801779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:06.801757 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" event={"ID":"07ebe837-5275-464d-861d-cc6825b54ef5","Type":"ContainerStarted","Data":"15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d"} Jul 23 00:51:07.224493 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:07.224379 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:51:07.224862 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:07.224839 2585 patch_prober.go:28] interesting pod/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 00:51:07.225057 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:07.225012 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 00:51:08.224453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:08.224414 2585 patch_prober.go:28] interesting pod/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 00:51:08.225090 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:08.225047 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 00:51:09.224752 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:09.224704 2585 patch_prober.go:28] interesting pod/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 00:51:09.225245 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:09.224772 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 00:51:10.224881 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:10.224829 2585 patch_prober.go:28] interesting pod/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 00:51:10.225400 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:10.224902 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 00:51:11.227521 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:11.227489 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:51:11.227928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:11.227744 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:51:11.228508 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:11.228487 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:51:13.502741 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:13.502686 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:17.991152 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:17.991116 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:17.997091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:17.997067 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm"] Jul 23 00:51:18.000377 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:18.000356 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89b85334-vzfxm"] Jul 23 00:51:18.384286 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:18.384177 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:19.412477 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:19.412427 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87362ca0-c02e-492d-9cee-896f45984347" path="/var/lib/kubelet/pods/87362ca0-c02e-492d-9cee-896f45984347/volumes" Jul 23 00:51:24.331225 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.331181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.433262 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.433229 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.592915 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.592827 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9"] Jul 23 00:51:24.593078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.593035 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" containerID="cri-o://15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d" gracePeriod=30 Jul 23 00:51:24.622799 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.622766 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.857283 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.857198 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.991087 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:24.991053 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:25.095960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:25.095929 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:25.240832 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:25.240801 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:26.230331 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:26.230288 2585 patch_prober.go:28] interesting pod/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:51:26.230722 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:26.230383 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:51:29.856828 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.856772 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:51:29.935733 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.935700 2585 generic.go:356] "Generic (PLEG): container finished" podID="07ebe837-5275-464d-861d-cc6825b54ef5" containerID="15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d" exitCode=0 Jul 23 00:51:29.935885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.935745 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" Jul 23 00:51:29.935885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.935779 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" event={"ID":"07ebe837-5275-464d-861d-cc6825b54ef5","Type":"ContainerDied","Data":"15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d"} Jul 23 00:51:29.935885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.935832 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9" event={"ID":"07ebe837-5275-464d-861d-cc6825b54ef5","Type":"ContainerDied","Data":"a111816f42573fc0ae1f72b07cc4a75ab2a0c87e4446b3470a08b6f64ba51ec4"} Jul 23 00:51:29.935885 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.935855 2585 scope.go:117] "RemoveContainer" containerID="15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d" Jul 23 00:51:29.944931 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.944915 2585 scope.go:117] "RemoveContainer" containerID="6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785" Jul 23 00:51:29.953118 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.953097 2585 scope.go:117] "RemoveContainer" containerID="15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d" Jul 23 00:51:29.953344 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:51:29.953329 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d\": container with ID starting with 15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d not found: ID does not exist" containerID="15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d" Jul 23 00:51:29.953399 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.953353 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d"} err="failed to get container status \"15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d\": rpc error: code = NotFound desc = could not find container \"15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d\": container with ID starting with 15946da6c76992933107fae34a4bbf6d9ab1e4c391e9ec876b00ae2c8288758d not found: ID does not exist" Jul 23 00:51:29.953399 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.953369 2585 scope.go:117] "RemoveContainer" containerID="6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785" Jul 23 00:51:29.953591 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:51:29.953568 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785\": container with ID starting with 6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785 not found: ID does not exist" containerID="6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785" Jul 23 00:51:29.953645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.953595 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785"} err="failed to get container status \"6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785\": rpc error: code = NotFound desc = could not find container \"6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785\": container with ID starting with 6533b10b20e62d2ac48fb53a7860c1fc5d70447d597a1e19abeb0e3a8a094785 not found: ID does not exist" Jul 23 00:51:29.957414 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957397 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-istio-token\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957481 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957436 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-data\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957481 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957453 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-credential-socket\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957553 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957503 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ebe837-5275-464d-861d-cc6825b54ef5-istio-podinfo\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957547 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6trz\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-kube-api-access-v6trz\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957576 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ebe837-5275-464d-861d-cc6825b54ef5-istiod-ca-cert\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957709 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957612 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-envoy\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957709 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957667 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-socket\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957812 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957711 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-certs\") pod \"07ebe837-5275-464d-861d-cc6825b54ef5\" (UID: \"07ebe837-5275-464d-861d-cc6825b54ef5\") " Jul 23 00:51:29.957867 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.957791 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:29.958036 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.958011 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:29.958149 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.958033 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:29.958149 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.958112 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:29.958456 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.958426 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ebe837-5275-464d-861d-cc6825b54ef5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:51:29.958808 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.958783 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-data" (OuterVolumeSpecName: "istio-data") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:29.959710 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.959681 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-istio-token" (OuterVolumeSpecName: "istio-token") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:29.959816 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.959738 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07ebe837-5275-464d-861d-cc6825b54ef5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:51:29.959816 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.959756 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-kube-api-access-v6trz" (OuterVolumeSpecName: "kube-api-access-v6trz") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "kube-api-access-v6trz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:29.959928 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:29.959830 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07ebe837-5275-464d-861d-cc6825b54ef5" (UID: "07ebe837-5275-464d-861d-cc6825b54ef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:30.058453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058425 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058453 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058451 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ebe837-5275-464d-861d-cc6825b54ef5-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058491 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6trz\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-kube-api-access-v6trz\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058506 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ebe837-5275-464d-861d-cc6825b54ef5-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058517 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058527 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058538 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ebe837-5275-464d-861d-cc6825b54ef5-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.058657 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.058550 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ebe837-5275-464d-861d-cc6825b54ef5-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:51:30.257398 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.257368 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9"] Jul 23 00:51:30.262284 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.262260 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-89b85334-openshift-default-d9dfb6595-875k9"] Jul 23 00:51:30.935832 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.935799 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:30.991924 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:30.991893 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.092610 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:31.092567 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.191363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:31.191276 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.325537 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:31.325491 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.412042 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:31.412014 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" path="/var/lib/kubelet/pods/07ebe837-5275-464d-861d-cc6825b54ef5/volumes" Jul 23 00:51:37.117913 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:37.117846 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:37.180761 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:37.180733 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:44.143340 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.143308 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:44.442600 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.442516 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj"] Jul 23 00:51:44.442995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.442977 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" Jul 23 00:51:44.443078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.442999 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" Jul 23 00:51:44.443078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443011 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87362ca0-c02e-492d-9cee-896f45984347" containerName="revoke-keys" Jul 23 00:51:44.443078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443020 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87362ca0-c02e-492d-9cee-896f45984347" containerName="revoke-keys" Jul 23 00:51:44.443078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443031 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" Jul 23 00:51:44.443078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443039 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" Jul 23 00:51:44.443321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443143 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87362ca0-c02e-492d-9cee-896f45984347" containerName="revoke-keys" Jul 23 00:51:44.443321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443158 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" Jul 23 00:51:44.443321 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.443169 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ebe837-5275-464d-861d-cc6825b54ef5" containerName="istio-proxy" Jul 23 00:51:44.445308 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.445288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.447445 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.447424 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3c46ebab-openshift-default-dockercfg-6qqvh\"" Jul 23 00:51:44.461367 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.461342 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj"] Jul 23 00:51:44.464211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464185 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-socket\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464382 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-credential-socket\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464639 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-data\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-certs\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-podinfo\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464703 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istiod-ca-cert\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-token\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.464774 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-envoy\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.465065 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.464784 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5q6\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-kube-api-access-wh5q6\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.468261 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.468239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:44.565810 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-data\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.565810 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565817 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-certs\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565833 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-podinfo\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565851 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istiod-ca-cert\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-token\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565889 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-envoy\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.565913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5q6\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-kube-api-access-wh5q6\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-socket\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566059 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-credential-socket\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566428 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-certs\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566428 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566392 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-data\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566752 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istiod-ca-cert\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566880 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566856 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-socket\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.566925 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.566902 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-credential-socket\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.568263 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.568234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-envoy\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.568586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.568568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-podinfo\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.573338 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.573316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-token\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.573429 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.573394 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5q6\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-kube-api-access-wh5q6\") pod \"e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.755968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.755939 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:44.900164 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.900137 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj"] Jul 23 00:51:44.902314 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:51:44.902244 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f29d5c_c854_4f8c_85ef_a79c52e2364e.slice/crio-0059d2248d6dd3136befe5234e3664d24e31ada97b592070b5c5fa9ea578cfdb WatchSource:0}: Error finding container 0059d2248d6dd3136befe5234e3664d24e31ada97b592070b5c5fa9ea578cfdb: Status 404 returned error can't find the container with id 0059d2248d6dd3136befe5234e3664d24e31ada97b592070b5c5fa9ea578cfdb Jul 23 00:51:44.904665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.904622 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:44.904779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.904698 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:44.904779 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:44.904745 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:45.021498 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:45.021396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" event={"ID":"89f29d5c-c854-4f8c-85ef-a79c52e2364e","Type":"ContainerStarted","Data":"32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14"} Jul 23 00:51:45.021498 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:45.021452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" event={"ID":"89f29d5c-c854-4f8c-85ef-a79c52e2364e","Type":"ContainerStarted","Data":"0059d2248d6dd3136befe5234e3664d24e31ada97b592070b5c5fa9ea578cfdb"} Jul 23 00:51:45.046772 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:45.046722 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podStartSLOduration=1.046706205 podStartE2EDuration="1.046706205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:51:45.045306752 +0000 UTC m=+1934.342001721" watchObservedRunningTime="2026-07-23 00:51:45.046706205 +0000 UTC m=+1934.343401199" Jul 23 00:51:45.570667 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:45.570622 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:45.756973 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:45.756944 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:45.761907 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:45.761881 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:46.027611 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:46.027586 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:46.028585 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:46.028564 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:51:47.036425 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:47.036396 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:47.150809 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:47.150774 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:47.242915 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:47.242872 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:47.448833 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:47.448753 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:47.767296 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:47.767269 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:47.867631 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:47.867599 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:51.586590 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:51.586558 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:58.922003 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:51:58.921966 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:01.785360 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:01.784987 2585 patch_prober.go:28] interesting pod/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:55638->10.132.0.63:15021: read: connection reset by peer" start-of-body= Jul 23 00:52:01.785360 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:01.785054 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:55638->10.132.0.63:15021: read: connection reset by peer" Jul 23 00:52:02.118626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:02.118539 2585 generic.go:356] "Generic (PLEG): container finished" podID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerID="32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14" exitCode=0 Jul 23 00:52:02.118797 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:02.118619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" event={"ID":"89f29d5c-c854-4f8c-85ef-a79c52e2364e","Type":"ContainerDied","Data":"32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14"} Jul 23 00:52:02.119063 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:02.119047 2585 scope.go:117] "RemoveContainer" containerID="32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14" Jul 23 00:52:02.120024 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:02.120001 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:52:02.120083 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:02.120063 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:52:02.120119 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:02.120095 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:52:03.129896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:03.129851 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" event={"ID":"89f29d5c-c854-4f8c-85ef-a79c52e2364e","Type":"ContainerStarted","Data":"88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea"} Jul 23 00:52:03.757215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:03.757182 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:52:03.757420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:03.757398 2585 patch_prober.go:28] interesting pod/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 00:52:03.757481 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:03.757448 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 00:52:04.757191 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:04.757153 2585 patch_prober.go:28] interesting pod/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 00:52:04.757601 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:04.757219 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 00:52:05.756542 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:05.756499 2585 patch_prober.go:28] interesting pod/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 00:52:05.756754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:05.756572 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 00:52:05.922982 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:05.922939 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:06.757094 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:06.757058 2585 patch_prober.go:28] interesting pod/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 00:52:06.757290 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:06.757123 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 00:52:07.178675 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:07.178592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:07.759624 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:07.759587 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:52:07.759844 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:07.759823 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:52:07.760591 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:07.760570 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:52:18.957614 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:18.957582 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:19.058042 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:19.058016 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:30.384449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.384369 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms"] Jul 23 00:52:30.393209 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.393000 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:30.395357 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.395328 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:52:30.395514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.395395 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:52:30.395946 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.395921 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms"] Jul 23 00:52:30.476192 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.476157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3247ab0-4555-4ac5-92df-affa70ecdd64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms\" (UID: \"e3247ab0-4555-4ac5-92df-affa70ecdd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:30.576831 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.576799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3247ab0-4555-4ac5-92df-affa70ecdd64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms\" (UID: \"e3247ab0-4555-4ac5-92df-affa70ecdd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:30.577381 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.577362 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3247ab0-4555-4ac5-92df-affa70ecdd64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms\" (UID: \"e3247ab0-4555-4ac5-92df-affa70ecdd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:30.706287 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.706197 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:30.838624 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:30.838599 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms"] Jul 23 00:52:30.840689 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:52:30.840655 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3247ab0_4555_4ac5_92df_affa70ecdd64.slice/crio-7fb4648bfe10396e1443bd9d71ab11ac0e83d502eff5d14e8291065a2ea6975e WatchSource:0}: Error finding container 7fb4648bfe10396e1443bd9d71ab11ac0e83d502eff5d14e8291065a2ea6975e: Status 404 returned error can't find the container with id 7fb4648bfe10396e1443bd9d71ab11ac0e83d502eff5d14e8291065a2ea6975e Jul 23 00:52:31.281004 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:31.280924 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" event={"ID":"e3247ab0-4555-4ac5-92df-affa70ecdd64","Type":"ContainerStarted","Data":"4eefc34c52d4d90b14d8ae4d456104489434de9784b6d57b5a8913ea1c4e97f9"} Jul 23 00:52:31.281004 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:31.280983 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" event={"ID":"e3247ab0-4555-4ac5-92df-affa70ecdd64","Type":"ContainerStarted","Data":"7fb4648bfe10396e1443bd9d71ab11ac0e83d502eff5d14e8291065a2ea6975e"} Jul 23 00:52:31.296281 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:31.296240 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" podStartSLOduration=1.296225916 podStartE2EDuration="1.296225916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:52:31.293921188 +0000 UTC m=+1980.590616168" watchObservedRunningTime="2026-07-23 00:52:31.296225916 +0000 UTC m=+1980.592920883" Jul 23 00:52:32.287277 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:32.287243 2585 generic.go:356] "Generic (PLEG): container finished" podID="e3247ab0-4555-4ac5-92df-affa70ecdd64" containerID="4eefc34c52d4d90b14d8ae4d456104489434de9784b6d57b5a8913ea1c4e97f9" exitCode=0 Jul 23 00:52:32.287671 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:32.287310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" event={"ID":"e3247ab0-4555-4ac5-92df-affa70ecdd64","Type":"ContainerDied","Data":"4eefc34c52d4d90b14d8ae4d456104489434de9784b6d57b5a8913ea1c4e97f9"} Jul 23 00:52:33.429498 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:33.429450 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:33.603365 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:33.603281 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3247ab0-4555-4ac5-92df-affa70ecdd64-maas-api-service-ca\") pod \"e3247ab0-4555-4ac5-92df-affa70ecdd64\" (UID: \"e3247ab0-4555-4ac5-92df-affa70ecdd64\") " Jul 23 00:52:33.603679 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:33.603652 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3247ab0-4555-4ac5-92df-affa70ecdd64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3247ab0-4555-4ac5-92df-affa70ecdd64" (UID: "e3247ab0-4555-4ac5-92df-affa70ecdd64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:52:33.704846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:33.704812 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3247ab0-4555-4ac5-92df-affa70ecdd64-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:52:34.298953 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:34.298926 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" Jul 23 00:52:34.299126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:34.298967 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms" event={"ID":"e3247ab0-4555-4ac5-92df-affa70ecdd64","Type":"ContainerDied","Data":"7fb4648bfe10396e1443bd9d71ab11ac0e83d502eff5d14e8291065a2ea6975e"} Jul 23 00:52:34.299126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:34.299017 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb4648bfe10396e1443bd9d71ab11ac0e83d502eff5d14e8291065a2ea6975e" Jul 23 00:52:45.795393 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:45.795360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:50.460187 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:50.460143 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms"] Jul 23 00:52:50.461995 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:50.461974 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3c46ebab-lzdms"] Jul 23 00:52:50.465602 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:50.465581 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:51.412938 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:51.412908 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3247ab0-4555-4ac5-92df-affa70ecdd64" path="/var/lib/kubelet/pods/e3247ab0-4555-4ac5-92df-affa70ecdd64/volumes" Jul 23 00:52:52.836233 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:52.836199 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:57.906764 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:52:57.906733 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:31.428343 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:31.428309 2585 scope.go:117] "RemoveContainer" containerID="b91ca61b96fac1398c9162e4dcc0d194f17ef75212993fbdaf94045e01f74c04" Jul 23 00:53:31.436781 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:31.436761 2585 scope.go:117] "RemoveContainer" containerID="d189e32853856965adaf6497079ca653a95424b590db89bede0228dffcb81ca8" Jul 23 00:53:34.051870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:34.051836 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:35.042244 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:35.042211 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:40.467830 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:40.467788 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:40.567645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:40.567612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:40.758705 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:40.758616 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj"] Jul 23 00:53:40.758960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:40.758890 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" containerID="cri-o://88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea" gracePeriod=30 Jul 23 00:53:40.790068 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:40.790024 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:41.044895 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:41.044813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:42.238896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:42.238863 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:42.344989 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:42.344958 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:42.438140 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:42.438107 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:43.294628 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:43.294581 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:43.389072 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:43.389036 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:43.488525 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:43.488459 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:45.010704 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:45.010666 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:46.021040 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.020980 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:53:46.156022 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.155992 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-data\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156033 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-envoy\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156075 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-certs\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156098 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-socket\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156132 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-token\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156177 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-credential-socket\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156211 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156205 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-podinfo\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156571 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156234 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh5q6\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-kube-api-access-wh5q6\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156571 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156262 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istiod-ca-cert\") pod \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\" (UID: \"89f29d5c-c854-4f8c-85ef-a79c52e2364e\") " Jul 23 00:53:46.156571 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156398 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:53:46.156571 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156423 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:53:46.156792 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156631 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.156792 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156651 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.156792 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156710 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:53:46.156919 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156808 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:53:46.156994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.156974 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-data" (OuterVolumeSpecName: "istio-data") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:53:46.158918 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.158882 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:53:46.159034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.158923 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:53:46.159034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.158943 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-kube-api-access-wh5q6" (OuterVolumeSpecName: "kube-api-access-wh5q6") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "kube-api-access-wh5q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:53:46.159034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.158970 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-token" (OuterVolumeSpecName: "istio-token") pod "89f29d5c-c854-4f8c-85ef-a79c52e2364e" (UID: "89f29d5c-c854-4f8c-85ef-a79c52e2364e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:53:46.257232 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257207 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.257232 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257229 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.257232 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257238 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5q6\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-kube-api-access-wh5q6\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.257406 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257247 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.257406 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257256 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.257406 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257263 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.257406 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.257270 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f29d5c-c854-4f8c-85ef-a79c52e2364e-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:53:46.696593 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.696558 2585 generic.go:356] "Generic (PLEG): container finished" podID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerID="88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea" exitCode=0 Jul 23 00:53:46.696593 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.696601 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" Jul 23 00:53:46.696802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.696639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" event={"ID":"89f29d5c-c854-4f8c-85ef-a79c52e2364e","Type":"ContainerDied","Data":"88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea"} Jul 23 00:53:46.696802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.696690 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj" event={"ID":"89f29d5c-c854-4f8c-85ef-a79c52e2364e","Type":"ContainerDied","Data":"0059d2248d6dd3136befe5234e3664d24e31ada97b592070b5c5fa9ea578cfdb"} Jul 23 00:53:46.696802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.696711 2585 scope.go:117] "RemoveContainer" containerID="88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea" Jul 23 00:53:46.706954 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.706933 2585 scope.go:117] "RemoveContainer" containerID="32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14" Jul 23 00:53:46.716286 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.716266 2585 scope.go:117] "RemoveContainer" containerID="88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea" Jul 23 00:53:46.716556 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:53:46.716538 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea\": container with ID starting with 88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea not found: ID does not exist" containerID="88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea" Jul 23 00:53:46.716806 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.716566 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea"} err="failed to get container status \"88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea\": rpc error: code = NotFound desc = could not find container \"88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea\": container with ID starting with 88702ff0c0e6c13f70937e84a79a316264e190a9f47be9007d5abc89f49f9dea not found: ID does not exist" Jul 23 00:53:46.716806 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.716584 2585 scope.go:117] "RemoveContainer" containerID="32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14" Jul 23 00:53:46.716991 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:53:46.716838 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14\": container with ID starting with 32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14 not found: ID does not exist" containerID="32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14" Jul 23 00:53:46.716991 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.716861 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14"} err="failed to get container status \"32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14\": rpc error: code = NotFound desc = could not find container \"32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14\": container with ID starting with 32723fc243bf9555917a68c97a51ce88e0058d6f339f532181f8b2f72bb5ae14 not found: ID does not exist" Jul 23 00:53:46.719443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.719422 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj"] Jul 23 00:53:46.724210 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:46.724190 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3c46ebab-openshift-default-577b596687-zjqbj"] Jul 23 00:53:47.411735 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:47.411701 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" path="/var/lib/kubelet/pods/89f29d5c-c854-4f8c-85ef-a79c52e2364e/volumes" Jul 23 00:53:50.817242 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:50.817212 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:50.917687 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:50.917654 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:51.523122 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:51.523072 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:57.551705 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:57.551622 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:57.669859 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:57.669823 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:57.841922 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:53:57.841838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:00.143210 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:00.143178 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:08.370960 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:08.370918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:08.756749 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:08.756718 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:14.885896 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:14.885858 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:29.818925 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:29.818891 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:31.313389 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:31.313362 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:54:31.316572 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:31.316545 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:54:31.461526 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:31.461497 2585 scope.go:117] "RemoveContainer" containerID="0f8a1afc73f240cc82a0fe8c1c02ae3bca0a66b26477b10a00010c0047a56049" Jul 23 00:54:35.902660 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:35.902619 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.005509 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.005476 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.295295 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.295262 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.599487 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599395 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr"] Jul 23 00:54:36.599809 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599795 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" Jul 23 00:54:36.599852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599811 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" Jul 23 00:54:36.599852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599825 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" Jul 23 00:54:36.599852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599831 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" Jul 23 00:54:36.599852 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599850 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3247ab0-4555-4ac5-92df-affa70ecdd64" containerName="revoke-keys" Jul 23 00:54:36.599979 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599857 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3247ab0-4555-4ac5-92df-affa70ecdd64" containerName="revoke-keys" Jul 23 00:54:36.599979 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599917 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" Jul 23 00:54:36.599979 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599924 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3247ab0-4555-4ac5-92df-affa70ecdd64" containerName="revoke-keys" Jul 23 00:54:36.599979 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.599930 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f29d5c-c854-4f8c-85ef-a79c52e2364e" containerName="istio-proxy" Jul 23 00:54:36.603089 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.603067 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.607126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.607108 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-551ea4-openshift-default-dockercfg-jkb6j\"" Jul 23 00:54:36.617112 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.617089 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr"] Jul 23 00:54:36.626116 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.626091 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.681744 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681714 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-credential-socket\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.681892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681750 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-envoy\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.681892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681804 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-data\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.681892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68xg\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-kube-api-access-q68xg\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.681892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c24b23-c238-45e5-b656-727531f233b3-istio-podinfo\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.682091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c24b23-c238-45e5-b656-727531f233b3-istiod-ca-cert\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.682091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681917 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-socket\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.682091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681960 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-certs\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.682091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.681982 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-istio-token\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.782856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.782827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68xg\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-kube-api-access-q68xg\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.782856 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.782861 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c24b23-c238-45e5-b656-727531f233b3-istio-podinfo\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.782879 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c24b23-c238-45e5-b656-727531f233b3-istiod-ca-cert\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.782897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-socket\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783004 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-certs\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783173 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-istio-token\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-credential-socket\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-envoy\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-data\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783610 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-certs\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783670 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783614 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-credential-socket\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783729 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-data\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783729 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783700 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c24b23-c238-45e5-b656-727531f233b3-istiod-ca-cert\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.783957 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.783930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-socket\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.785497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.785456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-envoy\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.785626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.785611 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c24b23-c238-45e5-b656-727531f233b3-istio-podinfo\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.791397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.791366 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68xg\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-kube-api-access-q68xg\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.791531 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.791456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-istio-token\") pod \"e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:36.917721 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:36.917622 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:37.068344 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.068319 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr"] Jul 23 00:54:37.070267 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:54:37.070237 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c24b23_c238_45e5_b656_727531f233b3.slice/crio-a7ea8331f4fb02a72b32758acbcc643d9700a03d8dfbde694c9bc79ed18fabb7 WatchSource:0}: Error finding container a7ea8331f4fb02a72b32758acbcc643d9700a03d8dfbde694c9bc79ed18fabb7: Status 404 returned error can't find the container with id a7ea8331f4fb02a72b32758acbcc643d9700a03d8dfbde694c9bc79ed18fabb7 Jul 23 00:54:37.072332 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.072302 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:37.072400 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.072368 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:37.072435 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.072399 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:37.275390 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.275354 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:37.966443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.966398 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" event={"ID":"58c24b23-c238-45e5-b656-727531f233b3","Type":"ContainerStarted","Data":"f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3"} Jul 23 00:54:37.966821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.966490 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" event={"ID":"58c24b23-c238-45e5-b656-727531f233b3","Type":"ContainerStarted","Data":"a7ea8331f4fb02a72b32758acbcc643d9700a03d8dfbde694c9bc79ed18fabb7"} Jul 23 00:54:37.987012 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:37.986967 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" podStartSLOduration=1.986953373 podStartE2EDuration="1.986953373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:54:37.986217526 +0000 UTC m=+2107.282912494" watchObservedRunningTime="2026-07-23 00:54:37.986953373 +0000 UTC m=+2107.283648341" Jul 23 00:54:38.918304 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:38.918271 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:38.923038 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:38.923013 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:38.971497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:38.971455 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:38.972451 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:38.972432 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:39.973542 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:39.973511 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:40.101174 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:40.101130 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:40.201949 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:40.201913 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:40.533380 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:40.533337 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:40.632745 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:40.632715 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:40.735424 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:40.735390 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:44.538772 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:44.538731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:44.931684 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:44.931561 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:45.221776 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.221690 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p"] Jul 23 00:54:45.233495 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.226448 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.235110 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.235078 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fb82e4-openshift-default-dockercfg-jcdxz\"" Jul 23 00:54:45.250058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.250030 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p"] Jul 23 00:54:45.251880 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.251855 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:45.366742 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.366709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f8c9b4-2161-4f30-85a0-438705500426-istio-podinfo\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.366927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.366753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-credential-socket\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.366927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.366780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-socket\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.366927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.366824 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-certs\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.366927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.366893 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-istio-token\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.366927 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.366925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-envoy\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.367098 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.367020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-data\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.367132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.367105 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f8c9b4-2161-4f30-85a0-438705500426-istiod-ca-cert\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.367186 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.367171 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56d2\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-kube-api-access-b56d2\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.459739 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.459711 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:45.467913 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.467890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-credential-socket\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.467990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.467928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-socket\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.467990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.467945 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-certs\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.467990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.467971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-istio-token\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-envoy\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468026 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-data\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f8c9b4-2161-4f30-85a0-438705500426-istiod-ca-cert\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56d2\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-kube-api-access-b56d2\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f8c9b4-2161-4f30-85a0-438705500426-istio-podinfo\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-certs\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468345 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-socket\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468507 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468375 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-credential-socket\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468638 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468612 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-data\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.468937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.468918 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f8c9b4-2161-4f30-85a0-438705500426-istiod-ca-cert\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.470585 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.470558 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-envoy\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.470838 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.470820 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f8c9b4-2161-4f30-85a0-438705500426-istio-podinfo\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.475149 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.475092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-istio-token\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.475228 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.475206 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56d2\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-kube-api-access-b56d2\") pod \"e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.551764 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.551731 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:45.734771 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.734745 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p"] Jul 23 00:54:45.737202 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:54:45.737178 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f8c9b4_2161_4f30_85a0_438705500426.slice/crio-ec02dd96bf2dab80032ce16953af3effca09847732e895dea517fd54d3e0a2f5 WatchSource:0}: Error finding container ec02dd96bf2dab80032ce16953af3effca09847732e895dea517fd54d3e0a2f5: Status 404 returned error can't find the container with id ec02dd96bf2dab80032ce16953af3effca09847732e895dea517fd54d3e0a2f5 Jul 23 00:54:45.739413 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.739383 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:45.739549 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.739454 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:45.739625 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:45.739590 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:46.018778 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:46.017739 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" event={"ID":"46f8c9b4-2161-4f30-85a0-438705500426","Type":"ContainerStarted","Data":"7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90"} Jul 23 00:54:46.018778 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:46.017803 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" event={"ID":"46f8c9b4-2161-4f30-85a0-438705500426","Type":"ContainerStarted","Data":"ec02dd96bf2dab80032ce16953af3effca09847732e895dea517fd54d3e0a2f5"} Jul 23 00:54:46.038958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:46.038890 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" podStartSLOduration=1.0388694 podStartE2EDuration="1.0388694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:54:46.035352638 +0000 UTC m=+2115.332047606" watchObservedRunningTime="2026-07-23 00:54:46.0388694 +0000 UTC m=+2115.335564369" Jul 23 00:54:46.552813 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:46.552779 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:46.557992 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:46.557967 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:47.027307 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:47.027271 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:47.032094 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:47.032069 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:54:48.026033 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:48.025930 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:48.125296 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:48.125261 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:48.233089 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:48.233057 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:48.421707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:48.421620 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:48.591202 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:48.591171 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:48.692571 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:48.692491 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:52.057201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.057167 2585 generic.go:356] "Generic (PLEG): container finished" podID="58c24b23-c238-45e5-b656-727531f233b3" containerID="f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3" exitCode=0 Jul 23 00:54:52.057636 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.057247 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" event={"ID":"58c24b23-c238-45e5-b656-727531f233b3","Type":"ContainerDied","Data":"f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3"} Jul 23 00:54:52.057721 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.057704 2585 scope.go:117] "RemoveContainer" containerID="f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3" Jul 23 00:54:52.058711 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.058684 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:52.058771 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.058764 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:52.058810 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.058800 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:54:52.593803 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:52.593766 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:53.064514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:53.064473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" event={"ID":"58c24b23-c238-45e5-b656-727531f233b3","Type":"ContainerStarted","Data":"81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55"} Jul 23 00:54:53.918715 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:53.918617 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:53.920398 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:53.920075 2585 patch_prober.go:28] interesting pod/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 00:54:53.920398 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:53.920130 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 00:54:53.971984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:53.971944 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:54.918875 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:54.918829 2585 patch_prober.go:28] interesting pod/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 00:54:54.919389 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:54.918899 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 00:54:55.927177 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:55.927141 2585 patch_prober.go:28] interesting pod/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 00:54:55.927772 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:55.927738 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 00:54:56.927840 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:56.927806 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:54:56.928891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:54:56.928860 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:55:00.108917 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:00.108878 2585 generic.go:356] "Generic (PLEG): container finished" podID="46f8c9b4-2161-4f30-85a0-438705500426" containerID="7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90" exitCode=0 Jul 23 00:55:00.109371 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:00.108958 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" event={"ID":"46f8c9b4-2161-4f30-85a0-438705500426","Type":"ContainerDied","Data":"7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90"} Jul 23 00:55:00.109506 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:00.109482 2585 scope.go:117] "RemoveContainer" containerID="7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90" Jul 23 00:55:00.110652 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:00.110622 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:55:00.110786 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:00.110707 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:55:00.110786 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:00.110759 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:55:01.121970 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:01.121925 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" event={"ID":"46f8c9b4-2161-4f30-85a0-438705500426","Type":"ContainerStarted","Data":"9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518"} Jul 23 00:55:01.373596 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:01.373505 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:01.473233 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:01.473151 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:01.552146 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:01.552109 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:55:01.553392 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:01.553367 2585 patch_prober.go:28] interesting pod/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 00:55:01.553500 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:01.553419 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 00:55:02.027231 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:02.027192 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:55:02.552060 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:02.552021 2585 patch_prober.go:28] interesting pod/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 00:55:02.552510 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:02.552083 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 00:55:03.553085 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:03.553045 2585 patch_prober.go:28] interesting pod/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 00:55:03.553490 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:03.553101 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 00:55:04.552118 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:04.552079 2585 patch_prober.go:28] interesting pod/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 00:55:04.552297 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:04.552143 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 00:55:05.557092 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:05.557066 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:55:05.558096 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:05.558075 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:55:09.421984 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:09.421873 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:09.522616 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:09.522587 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:15.894406 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:15.894370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:31.487910 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:31.487841 2585 scope.go:117] "RemoveContainer" containerID="a921be5f63b8fdb813fb0782413ba500418481a55508e7d6ec0c59862a7b13c7" Jul 23 00:55:44.102607 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:55:44.102569 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:23.184349 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:23.184317 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:24.236077 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.236041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:24.325664 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.325632 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86"] Jul 23 00:56:24.329161 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.329146 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:24.331397 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.331377 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:56:24.331543 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.331398 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:56:24.338990 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.338959 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86"] Jul 23 00:56:24.469710 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.469677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86\" (UID: \"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:24.570631 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.570536 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86\" (UID: \"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:24.571204 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.571184 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86\" (UID: \"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:24.639887 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.639845 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:24.774914 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.774832 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86"] Jul 23 00:56:24.778076 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:56:24.778049 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa33fcc_f1e0_4af6_a8aa_1fd11d7f2780.slice/crio-00d3b6b9ef2b51b11de24438e8f5c9c06707d419940e190245dc3be470f1f4b4 WatchSource:0}: Error finding container 00d3b6b9ef2b51b11de24438e8f5c9c06707d419940e190245dc3be470f1f4b4: Status 404 returned error can't find the container with id 00d3b6b9ef2b51b11de24438e8f5c9c06707d419940e190245dc3be470f1f4b4 Jul 23 00:56:24.915566 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:24.915492 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:25.021617 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:25.021581 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:25.588050 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:25.588011 2585 generic.go:356] "Generic (PLEG): container finished" podID="1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780" containerID="96b7dd94705f19c8c1a63374248f4aeaaba1619938a4af659d19af3c00a24f1c" exitCode=0 Jul 23 00:56:25.588426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:25.588065 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" event={"ID":"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780","Type":"ContainerDied","Data":"96b7dd94705f19c8c1a63374248f4aeaaba1619938a4af659d19af3c00a24f1c"} Jul 23 00:56:25.588426 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:25.588122 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" event={"ID":"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780","Type":"ContainerStarted","Data":"00d3b6b9ef2b51b11de24438e8f5c9c06707d419940e190245dc3be470f1f4b4"} Jul 23 00:56:26.734816 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:26.734793 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:26.790323 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:26.790291 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780-maas-api-service-ca\") pod \"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780\" (UID: \"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780\") " Jul 23 00:56:26.790710 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:26.790685 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780" (UID: "1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:56:26.891069 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:26.890977 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:27.600309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:27.600277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" event={"ID":"1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780","Type":"ContainerDied","Data":"00d3b6b9ef2b51b11de24438e8f5c9c06707d419940e190245dc3be470f1f4b4"} Jul 23 00:56:27.600309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:27.600307 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86" Jul 23 00:56:27.600572 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:27.600326 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00d3b6b9ef2b51b11de24438e8f5c9c06707d419940e190245dc3be470f1f4b4" Jul 23 00:56:29.495693 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:29.495654 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:29.549547 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:29.549514 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:34.836805 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:34.836774 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:39.371201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:39.371162 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:39.383118 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:39.383087 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86"] Jul 23 00:56:39.385031 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:39.385011 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-551ea4-zhq86"] Jul 23 00:56:39.411839 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:39.411803 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780" path="/var/lib/kubelet/pods/1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780/volumes" Jul 23 00:56:39.793667 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:39.793626 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:45.951983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:45.951943 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:46.010512 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.010477 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:46.208223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.208124 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr"] Jul 23 00:56:46.208443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.208416 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" containerID="cri-o://81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55" gracePeriod=30 Jul 23 00:56:46.228322 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.228294 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:46.483571 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.483539 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:46.599048 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.599014 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h"] Jul 23 00:56:46.599422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.599410 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780" containerName="revoke-keys" Jul 23 00:56:46.599480 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.599423 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780" containerName="revoke-keys" Jul 23 00:56:46.599514 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.599506 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fa33fcc-f1e0-4af6-a8aa-1fd11d7f2780" containerName="revoke-keys" Jul 23 00:56:46.602720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.602701 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:46.604459 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.604440 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:56:46.604459 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.604442 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:56:46.608613 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.608592 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h"] Jul 23 00:56:46.776268 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.776163 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h\" (UID: \"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:46.877345 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.877308 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h\" (UID: \"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:46.877976 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.877956 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h\" (UID: \"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:46.916266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:46.916231 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:47.056802 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:47.056773 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h"] Jul 23 00:56:47.059200 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:56:47.059163 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25d9bdb_e75a_40fa_8ff0_cddefdd53f5b.slice/crio-1bec9b13fef4e66997ba26edba458149e7045c0cffafe9c5a6cd850f67a7110f WatchSource:0}: Error finding container 1bec9b13fef4e66997ba26edba458149e7045c0cffafe9c5a6cd850f67a7110f: Status 404 returned error can't find the container with id 1bec9b13fef4e66997ba26edba458149e7045c0cffafe9c5a6cd850f67a7110f Jul 23 00:56:47.304505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:47.304454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:47.404139 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:47.404106 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:47.709314 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:47.709277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" event={"ID":"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b","Type":"ContainerStarted","Data":"c285df8cf8badf0a9e52780859f20c899701b329758055d4c74cb13b7cd4ecfe"} Jul 23 00:56:47.709457 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:47.709336 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" event={"ID":"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b","Type":"ContainerStarted","Data":"1bec9b13fef4e66997ba26edba458149e7045c0cffafe9c5a6cd850f67a7110f"} Jul 23 00:56:47.723967 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:47.723920 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" podStartSLOduration=1.72390423 podStartE2EDuration="1.72390423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:56:47.72197277 +0000 UTC m=+2237.018667738" watchObservedRunningTime="2026-07-23 00:56:47.72390423 +0000 UTC m=+2237.020599198" Jul 23 00:56:48.716459 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:48.716425 2585 generic.go:356] "Generic (PLEG): container finished" podID="f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b" containerID="c285df8cf8badf0a9e52780859f20c899701b329758055d4c74cb13b7cd4ecfe" exitCode=0 Jul 23 00:56:48.716842 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:48.716509 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" event={"ID":"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b","Type":"ContainerDied","Data":"c285df8cf8badf0a9e52780859f20c899701b329758055d4c74cb13b7cd4ecfe"} Jul 23 00:56:49.864846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:49.864821 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:50.010641 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:50.010607 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b-maas-api-service-ca\") pod \"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b\" (UID: \"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b\") " Jul 23 00:56:50.010978 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:50.010953 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b" (UID: "f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:56:50.112322 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:50.112285 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:50.730121 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:50.730093 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" Jul 23 00:56:50.730121 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:50.730114 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h" event={"ID":"f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b","Type":"ContainerDied","Data":"1bec9b13fef4e66997ba26edba458149e7045c0cffafe9c5a6cd850f67a7110f"} Jul 23 00:56:50.730326 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:50.730158 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bec9b13fef4e66997ba26edba458149e7045c0cffafe9c5a6cd850f67a7110f" Jul 23 00:56:51.482421 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.482392 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:56:51.624719 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624684 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q68xg\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-kube-api-access-q68xg\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.624719 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624722 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c24b23-c238-45e5-b656-727531f233b3-istiod-ca-cert\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.624968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624755 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c24b23-c238-45e5-b656-727531f233b3-istio-podinfo\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.624968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624809 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-socket\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.624968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624852 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-envoy\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.624968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-credential-socket\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.624968 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624961 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-istio-token\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.625236 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.624989 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-data\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.625236 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625012 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-certs\") pod \"58c24b23-c238-45e5-b656-727531f233b3\" (UID: \"58c24b23-c238-45e5-b656-727531f233b3\") " Jul 23 00:56:51.625236 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625121 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:56:51.625395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625262 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:56:51.625395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625296 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c24b23-c238-45e5-b656-727531f233b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:56:51.625395 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625317 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:56:51.625605 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625403 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.625605 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625415 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.625605 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625442 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c24b23-c238-45e5-b656-727531f233b3-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.625605 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.625450 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.626206 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.626182 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:56:51.627306 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.627250 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58c24b23-c238-45e5-b656-727531f233b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:56:51.627306 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.627262 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:56:51.627498 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.627454 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:56:51.627548 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.627480 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-kube-api-access-q68xg" (OuterVolumeSpecName: "kube-api-access-q68xg") pod "58c24b23-c238-45e5-b656-727531f233b3" (UID: "58c24b23-c238-45e5-b656-727531f233b3"). InnerVolumeSpecName "kube-api-access-q68xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:56:51.726794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.726759 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.726794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.726787 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.726794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.726796 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c24b23-c238-45e5-b656-727531f233b3-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.726794 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.726805 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q68xg\" (UniqueName: \"kubernetes.io/projected/58c24b23-c238-45e5-b656-727531f233b3-kube-api-access-q68xg\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.727052 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.726815 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c24b23-c238-45e5-b656-727531f233b3-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:56:51.738051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.737976 2585 generic.go:356] "Generic (PLEG): container finished" podID="58c24b23-c238-45e5-b656-727531f233b3" containerID="81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55" exitCode=0 Jul 23 00:56:51.738051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.738029 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" Jul 23 00:56:51.738201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.738051 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" event={"ID":"58c24b23-c238-45e5-b656-727531f233b3","Type":"ContainerDied","Data":"81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55"} Jul 23 00:56:51.738201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.738130 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr" event={"ID":"58c24b23-c238-45e5-b656-727531f233b3","Type":"ContainerDied","Data":"a7ea8331f4fb02a72b32758acbcc643d9700a03d8dfbde694c9bc79ed18fabb7"} Jul 23 00:56:51.738201 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.738153 2585 scope.go:117] "RemoveContainer" containerID="81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55" Jul 23 00:56:51.747911 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.747894 2585 scope.go:117] "RemoveContainer" containerID="f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3" Jul 23 00:56:51.756030 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.756012 2585 scope.go:117] "RemoveContainer" containerID="81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55" Jul 23 00:56:51.756283 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:56:51.756260 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55\": container with ID starting with 81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55 not found: ID does not exist" containerID="81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55" Jul 23 00:56:51.756348 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.756292 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55"} err="failed to get container status \"81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55\": rpc error: code = NotFound desc = could not find container \"81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55\": container with ID starting with 81d939c4b1e0bd830d8a915f475b9122a7699a83d444522a3f9bd4f4c640ef55 not found: ID does not exist" Jul 23 00:56:51.756348 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.756311 2585 scope.go:117] "RemoveContainer" containerID="f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3" Jul 23 00:56:51.756655 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:56:51.756638 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3\": container with ID starting with f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3 not found: ID does not exist" containerID="f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3" Jul 23 00:56:51.756706 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.756660 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3"} err="failed to get container status \"f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3\": rpc error: code = NotFound desc = could not find container \"f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3\": container with ID starting with f760e6ed620d403e1e4a2d7b0561d17442d9e038f3c5a26476278d9782910de3 not found: ID does not exist" Jul 23 00:56:51.760020 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.759999 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr"] Jul 23 00:56:51.764114 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:51.764094 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-551ea4-openshift-default-7dbbd79fdc-vkbcr"] Jul 23 00:56:53.411365 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:53.411335 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c24b23-c238-45e5-b656-727531f233b3" path="/var/lib/kubelet/pods/58c24b23-c238-45e5-b656-727531f233b3/volumes" Jul 23 00:56:57.006412 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:56:57.006331 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:01.668270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:01.668234 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h"] Jul 23 00:57:01.671308 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:01.671284 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb82e4-shb8h"] Jul 23 00:57:01.673585 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:01.673560 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:02.097078 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:02.097033 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:03.412643 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:03.412610 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b" path="/var/lib/kubelet/pods/f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b/volumes" Jul 23 00:57:07.865003 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:07.864974 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:08.125740 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:08.125642 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p"] Jul 23 00:57:08.125902 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:08.125880 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" containerID="cri-o://9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518" gracePeriod=30 Jul 23 00:57:09.096322 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.096289 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77"] Jul 23 00:57:09.096965 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.096939 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" Jul 23 00:57:09.096965 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.096967 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" Jul 23 00:57:09.097126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.096980 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" Jul 23 00:57:09.097126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.096988 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" Jul 23 00:57:09.097126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.096998 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b" containerName="revoke-keys" Jul 23 00:57:09.097126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.097006 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b" containerName="revoke-keys" Jul 23 00:57:09.097126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.097105 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25d9bdb-e75a-40fa-8ff0-cddefdd53f5b" containerName="revoke-keys" Jul 23 00:57:09.097126 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.097118 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" Jul 23 00:57:09.100348 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.100328 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.102324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.102301 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-23cf83-openshift-default-dockercfg-t274d\"" Jul 23 00:57:09.111441 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.111419 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77"] Jul 23 00:57:09.185915 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.185882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-socket\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.185915 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.185917 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-data\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186176 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.185941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08a865-b615-4d5f-9cba-a269574320bf-istiod-ca-cert\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186176 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.186089 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08a865-b615-4d5f-9cba-a269574320bf-istio-podinfo\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186176 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.186121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-certs\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.186201 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-envoy\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.186231 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-istio-token\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.186256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zshz\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-kube-api-access-4zshz\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.186342 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.186324 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-credential-socket\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.287861 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.287825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-envoy\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.287861 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.287860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-istio-token\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.287876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zshz\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-kube-api-access-4zshz\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.287967 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-credential-socket\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-socket\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288050 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-data\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288127 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288089 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08a865-b615-4d5f-9cba-a269574320bf-istiod-ca-cert\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288394 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08a865-b615-4d5f-9cba-a269574320bf-istio-podinfo\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288394 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-certs\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288536 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-credential-socket\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288597 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288562 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-socket\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288644 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288591 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-data\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288734 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288712 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-certs\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.288958 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.288941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08a865-b615-4d5f-9cba-a269574320bf-istiod-ca-cert\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.290314 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.290295 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-envoy\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.290587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.290568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08a865-b615-4d5f-9cba-a269574320bf-istio-podinfo\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.295633 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.295609 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zshz\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-kube-api-access-4zshz\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.295633 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.295628 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-istio-token\") pod \"e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.307633 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.307595 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:09.374443 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.374332 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:09.414562 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.414532 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:09.555754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.555728 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77"] Jul 23 00:57:09.557831 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:57:09.557807 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee08a865_b615_4d5f_9cba_a269574320bf.slice/crio-ba74f864516a33815fad4314c342e7e35c4b3495d4f0d9503c599602032a9392 WatchSource:0}: Error finding container ba74f864516a33815fad4314c342e7e35c4b3495d4f0d9503c599602032a9392: Status 404 returned error can't find the container with id ba74f864516a33815fad4314c342e7e35c4b3495d4f0d9503c599602032a9392 Jul 23 00:57:09.560153 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.560119 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:09.560253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.560184 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:09.560253 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.560212 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:09.837246 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.837211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" event={"ID":"ee08a865-b615-4d5f-9cba-a269574320bf","Type":"ContainerStarted","Data":"270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8"} Jul 23 00:57:09.837390 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.837266 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" event={"ID":"ee08a865-b615-4d5f-9cba-a269574320bf","Type":"ContainerStarted","Data":"ba74f864516a33815fad4314c342e7e35c4b3495d4f0d9503c599602032a9392"} Jul 23 00:57:09.857892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:09.857839 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podStartSLOduration=0.857818833 podStartE2EDuration="857.818833ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:57:09.854061278 +0000 UTC m=+2259.150756264" watchObservedRunningTime="2026-07-23 00:57:09.857818833 +0000 UTC m=+2259.154513802" Jul 23 00:57:10.414914 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:10.414883 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:10.419673 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:10.419649 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:10.842830 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:10.842797 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:10.843716 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:10.843697 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:11.850736 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:11.850694 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:11.939777 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:11.939735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:12.010269 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:12.010232 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:12.476668 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:12.476622 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:12.606183 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:12.606102 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:12.709778 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:12.709719 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:13.385336 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.385311 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:57:13.530409 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530376 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-certs\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530632 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530444 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b56d2\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-kube-api-access-b56d2\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530632 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530527 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f8c9b4-2161-4f30-85a0-438705500426-istiod-ca-cert\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530632 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530557 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f8c9b4-2161-4f30-85a0-438705500426-istio-podinfo\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530632 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530595 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-credential-socket\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530640 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-envoy\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530674 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-socket\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530731 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-data\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.530847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530728 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:13.530847 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.530759 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-istio-token\") pod \"46f8c9b4-2161-4f30-85a0-438705500426\" (UID: \"46f8c9b4-2161-4f30-85a0-438705500426\") " Jul 23 00:57:13.531640 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.531155 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:13.531640 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.531509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:13.532007 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.531979 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-data" (OuterVolumeSpecName: "istio-data") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.532166 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f8c9b4-2161-4f30-85a0-438705500426-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533393 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533417 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f8c9b4-2161-4f30-85a0-438705500426-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533439 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533455 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533485 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533643 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f8c9b4-2161-4f30-85a0-438705500426-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:57:13.534563 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.533692 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:13.534991 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.534594 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-istio-token" (OuterVolumeSpecName: "istio-token") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:57:13.540240 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.540217 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-kube-api-access-b56d2" (OuterVolumeSpecName: "kube-api-access-b56d2") pod "46f8c9b4-2161-4f30-85a0-438705500426" (UID: "46f8c9b4-2161-4f30-85a0-438705500426"). InnerVolumeSpecName "kube-api-access-b56d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:57:13.634798 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.634747 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56d2\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-kube-api-access-b56d2\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.634798 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.634793 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f8c9b4-2161-4f30-85a0-438705500426-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.634798 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.634807 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f8c9b4-2161-4f30-85a0-438705500426-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.635035 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.634819 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f8c9b4-2161-4f30-85a0-438705500426-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:13.860204 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.860113 2585 generic.go:356] "Generic (PLEG): container finished" podID="46f8c9b4-2161-4f30-85a0-438705500426" containerID="9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518" exitCode=0 Jul 23 00:57:13.860204 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.860157 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" event={"ID":"46f8c9b4-2161-4f30-85a0-438705500426","Type":"ContainerDied","Data":"9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518"} Jul 23 00:57:13.860204 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.860170 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" Jul 23 00:57:13.860425 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.860235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p" event={"ID":"46f8c9b4-2161-4f30-85a0-438705500426","Type":"ContainerDied","Data":"ec02dd96bf2dab80032ce16953af3effca09847732e895dea517fd54d3e0a2f5"} Jul 23 00:57:13.860425 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.860260 2585 scope.go:117] "RemoveContainer" containerID="9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518" Jul 23 00:57:13.870278 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.870250 2585 scope.go:117] "RemoveContainer" containerID="7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90" Jul 23 00:57:13.879511 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.879487 2585 scope.go:117] "RemoveContainer" containerID="9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518" Jul 23 00:57:13.879791 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:57:13.879770 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518\": container with ID starting with 9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518 not found: ID does not exist" containerID="9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518" Jul 23 00:57:13.879840 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.879800 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518"} err="failed to get container status \"9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518\": rpc error: code = NotFound desc = could not find container \"9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518\": container with ID starting with 9865f45c6f1a88f1ac69ed12033bdb81c86cf53184bfb04409364871a4236518 not found: ID does not exist" Jul 23 00:57:13.879840 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.879818 2585 scope.go:117] "RemoveContainer" containerID="7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90" Jul 23 00:57:13.880064 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:57:13.880051 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90\": container with ID starting with 7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90 not found: ID does not exist" containerID="7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90" Jul 23 00:57:13.880106 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.880070 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90"} err="failed to get container status \"7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90\": rpc error: code = NotFound desc = could not find container \"7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90\": container with ID starting with 7d6c669a0552963c02da1232749406676bb5f0e67c5797aefa83c4bb539bab90 not found: ID does not exist" Jul 23 00:57:13.880615 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.880591 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p"] Jul 23 00:57:13.883111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:13.883089 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fb82e4-openshift-default-b8ff9c7-2gw2p"] Jul 23 00:57:15.412725 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:15.412685 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f8c9b4-2161-4f30-85a0-438705500426" path="/var/lib/kubelet/pods/46f8c9b4-2161-4f30-85a0-438705500426/volumes" Jul 23 00:57:16.375626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:16.375591 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:16.948309 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:16.948274 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.199567 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.199494 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n"] Jul 23 00:57:17.199897 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.199884 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" Jul 23 00:57:17.199938 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.199898 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" Jul 23 00:57:17.199938 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.199911 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" Jul 23 00:57:17.199938 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.199917 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" Jul 23 00:57:17.200075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.199993 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" Jul 23 00:57:17.200075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.200007 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c24b23-c238-45e5-b656-727531f233b3" containerName="istio-proxy" Jul 23 00:57:17.221487 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.217383 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.233496 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.224184 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n"] Jul 23 00:57:17.233496 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.224399 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e6fe2f-openshift-default-dockercfg-rm8vb\"" Jul 23 00:57:17.243105 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.243018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.368819 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.368789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5g5r\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-kube-api-access-k5g5r\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.368819 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.368823 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91cb4da9-1be6-446b-8922-c3cfa021cb72-istiod-ca-cert\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.368845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-socket\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.368907 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-envoy\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.368945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-token\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369091 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.369057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-podinfo\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.369107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-certs\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.369151 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-credential-socket\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.369254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.369197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-data\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470450 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-data\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470450 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470407 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5g5r\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-kube-api-access-k5g5r\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470450 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91cb4da9-1be6-446b-8922-c3cfa021cb72-istiod-ca-cert\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470450 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-socket\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-envoy\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-token\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470584 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-podinfo\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-certs\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.470807 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-credential-socket\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.471075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.470863 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-socket\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.471075 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.471010 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-data\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.471191 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.471171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-certs\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.471347 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.471319 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91cb4da9-1be6-446b-8922-c3cfa021cb72-istiod-ca-cert\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.471447 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.471354 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-credential-socket\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.472845 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.472824 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-envoy\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.473049 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.473032 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-podinfo\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.481286 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.481264 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-token\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.481437 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.481419 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5g5r\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-kube-api-access-k5g5r\") pod \"e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.496699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.496672 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.546284 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.546250 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:17.725490 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.725382 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n"] Jul 23 00:57:17.733578 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:57:17.733544 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cb4da9_1be6_446b_8922_c3cfa021cb72.slice/crio-66de57802d8e7c2a9ddbc2f2858823510b6735169b03e8cae89dbba478d921a0 WatchSource:0}: Error finding container 66de57802d8e7c2a9ddbc2f2858823510b6735169b03e8cae89dbba478d921a0: Status 404 returned error can't find the container with id 66de57802d8e7c2a9ddbc2f2858823510b6735169b03e8cae89dbba478d921a0 Jul 23 00:57:17.735712 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.735682 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:17.735796 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.735744 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:17.735836 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.735794 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:17.886255 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.886220 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" event={"ID":"91cb4da9-1be6-446b-8922-c3cfa021cb72","Type":"ContainerStarted","Data":"1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a"} Jul 23 00:57:17.886404 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.886280 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" event={"ID":"91cb4da9-1be6-446b-8922-c3cfa021cb72","Type":"ContainerStarted","Data":"66de57802d8e7c2a9ddbc2f2858823510b6735169b03e8cae89dbba478d921a0"} Jul 23 00:57:17.906793 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:17.906725 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podStartSLOduration=0.906704395 podStartE2EDuration="906.704395ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:57:17.904309627 +0000 UTC m=+2267.201004620" watchObservedRunningTime="2026-07-23 00:57:17.906704395 +0000 UTC m=+2267.203399365" Jul 23 00:57:18.546768 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:18.546737 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:18.551979 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:18.551949 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:18.891751 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:18.891671 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:18.892656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:18.892637 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:19.896094 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:19.896060 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:19.986272 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:19.986239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.119491 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:20.115879 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.523475 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:20.523428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.639435 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:20.639403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.741483 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:20.741434 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:23.923387 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:23.923357 2585 generic.go:356] "Generic (PLEG): container finished" podID="ee08a865-b615-4d5f-9cba-a269574320bf" containerID="270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8" exitCode=0 Jul 23 00:57:23.923916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:23.923429 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" event={"ID":"ee08a865-b615-4d5f-9cba-a269574320bf","Type":"ContainerDied","Data":"270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8"} Jul 23 00:57:23.923916 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:23.923847 2585 scope.go:117] "RemoveContainer" containerID="270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8" Jul 23 00:57:23.924869 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:23.924843 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:23.924934 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:23.924911 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:23.924992 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:23.924945 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:24.402673 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:24.402636 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:24.931818 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:24.931784 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" event={"ID":"ee08a865-b615-4d5f-9cba-a269574320bf","Type":"ContainerStarted","Data":"0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5"} Jul 23 00:57:25.415634 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:25.415602 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:25.416403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:25.416374 2585 patch_prober.go:28] interesting pod/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 23 00:57:25.416633 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:25.416601 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 23 00:57:25.843754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:25.843716 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:26.415962 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:26.415910 2585 patch_prober.go:28] interesting pod/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 23 00:57:26.416496 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:26.416002 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 23 00:57:27.415193 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:27.415159 2585 patch_prober.go:28] interesting pod/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 23 00:57:27.415557 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:27.415510 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 23 00:57:28.415892 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:28.415854 2585 patch_prober.go:28] interesting pod/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 23 00:57:28.416451 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:28.416416 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 23 00:57:29.425720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:29.425688 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:29.430339 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:29.430308 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:57:31.518645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:31.518614 2585 scope.go:117] "RemoveContainer" containerID="e9d95e8ba62d609a50c1accb2c00dc189332f9fe8c07465dbb6ad8149ee891c4" Jul 23 00:57:32.692952 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.692924 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj"] Jul 23 00:57:32.693404 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.693391 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f8c9b4-2161-4f30-85a0-438705500426" containerName="istio-proxy" Jul 23 00:57:32.698360 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.698342 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:32.700386 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.700343 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:57:32.700539 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.700389 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:57:32.705032 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.705004 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj"] Jul 23 00:57:32.817736 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.817701 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330c53f-9a5d-41d1-893c-6f9b0b70233f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23cf83-5njgj\" (UID: \"9330c53f-9a5d-41d1-893c-6f9b0b70233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:32.919058 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.919024 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330c53f-9a5d-41d1-893c-6f9b0b70233f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23cf83-5njgj\" (UID: \"9330c53f-9a5d-41d1-893c-6f9b0b70233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:32.919751 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:32.919726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330c53f-9a5d-41d1-893c-6f9b0b70233f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23cf83-5njgj\" (UID: \"9330c53f-9a5d-41d1-893c-6f9b0b70233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:33.013427 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.013394 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:33.162123 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.162101 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj"] Jul 23 00:57:33.164262 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:57:33.164236 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9330c53f_9a5d_41d1_893c_6f9b0b70233f.slice/crio-540e0ff2f1708df1ff66c29202a5f898918b22995562284b1348194f01671654 WatchSource:0}: Error finding container 540e0ff2f1708df1ff66c29202a5f898918b22995562284b1348194f01671654: Status 404 returned error can't find the container with id 540e0ff2f1708df1ff66c29202a5f898918b22995562284b1348194f01671654 Jul 23 00:57:33.170564 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.170503 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:33.272223 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.272146 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:33.893190 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.893149 2585 patch_prober.go:28] interesting pod/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 23 00:57:33.893595 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.893212 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 23 00:57:33.989403 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.989364 2585 generic.go:356] "Generic (PLEG): container finished" podID="9330c53f-9a5d-41d1-893c-6f9b0b70233f" containerID="4ab477183e28aaeea079e795849b06cf7e2827b13d767a7d3ac8a1a5a22538a6" exitCode=0 Jul 23 00:57:33.989584 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.989418 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" event={"ID":"9330c53f-9a5d-41d1-893c-6f9b0b70233f","Type":"ContainerDied","Data":"4ab477183e28aaeea079e795849b06cf7e2827b13d767a7d3ac8a1a5a22538a6"} Jul 23 00:57:33.989584 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.989483 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" event={"ID":"9330c53f-9a5d-41d1-893c-6f9b0b70233f","Type":"ContainerStarted","Data":"540e0ff2f1708df1ff66c29202a5f898918b22995562284b1348194f01671654"} Jul 23 00:57:33.991051 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.991029 2585 generic.go:356] "Generic (PLEG): container finished" podID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerID="1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a" exitCode=0 Jul 23 00:57:33.991183 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.991089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" event={"ID":"91cb4da9-1be6-446b-8922-c3cfa021cb72","Type":"ContainerDied","Data":"1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a"} Jul 23 00:57:33.991507 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.991492 2585 scope.go:117] "RemoveContainer" containerID="1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a" Jul 23 00:57:33.992646 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.992618 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:33.992720 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.992688 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:33.992758 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:33.992721 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:57:34.998256 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:34.998221 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" event={"ID":"91cb4da9-1be6-446b-8922-c3cfa021cb72","Type":"ContainerStarted","Data":"52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114"} Jul 23 00:57:35.169562 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.169535 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:35.242254 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.242218 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330c53f-9a5d-41d1-893c-6f9b0b70233f-maas-api-service-ca\") pod \"9330c53f-9a5d-41d1-893c-6f9b0b70233f\" (UID: \"9330c53f-9a5d-41d1-893c-6f9b0b70233f\") " Jul 23 00:57:35.242612 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.242584 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9330c53f-9a5d-41d1-893c-6f9b0b70233f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9330c53f-9a5d-41d1-893c-6f9b0b70233f" (UID: "9330c53f-9a5d-41d1-893c-6f9b0b70233f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:57:35.343595 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.343512 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330c53f-9a5d-41d1-893c-6f9b0b70233f-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:57:35.546821 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.546782 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:35.547034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.547010 2585 patch_prober.go:28] interesting pod/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 23 00:57:35.547087 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:35.547061 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 23 00:57:36.008976 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:36.008936 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" Jul 23 00:57:36.009490 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:36.009378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj" event={"ID":"9330c53f-9a5d-41d1-893c-6f9b0b70233f","Type":"ContainerDied","Data":"540e0ff2f1708df1ff66c29202a5f898918b22995562284b1348194f01671654"} Jul 23 00:57:36.009490 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:36.009425 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="540e0ff2f1708df1ff66c29202a5f898918b22995562284b1348194f01671654" Jul 23 00:57:36.547619 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:36.547577 2585 patch_prober.go:28] interesting pod/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 23 00:57:36.547843 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:36.547640 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 23 00:57:37.547662 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:37.547621 2585 patch_prober.go:28] interesting pod/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 23 00:57:37.548065 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:37.547681 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 23 00:57:38.547603 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:38.547562 2585 patch_prober.go:28] interesting pod/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 23 00:57:38.547777 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:38.547629 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 23 00:57:39.551622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:39.551588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:39.552214 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:39.551966 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:39.552694 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:39.552678 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:57:41.358756 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:41.358718 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:41.459085 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:41.459053 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:43.191988 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:43.191958 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:43.292622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:43.292589 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:48.442422 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:48.442384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:52.765157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:52.765117 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj"] Jul 23 00:57:52.767713 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:52.767686 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:52.767878 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:52.767861 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23cf83-5njgj"] Jul 23 00:57:53.180707 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:53.180625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:53.412372 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:53.412332 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9330c53f-9a5d-41d1-893c-6f9b0b70233f" path="/var/lib/kubelet/pods/9330c53f-9a5d-41d1-893c-6f9b0b70233f/volumes" Jul 23 00:57:58.909400 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:58.909367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:58.968065 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:58.968030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:59.169945 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.169863 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77"] Jul 23 00:57:59.170157 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.170131 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" containerID="cri-o://0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5" gracePeriod=30 Jul 23 00:57:59.189504 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.189451 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:59.425834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.425752 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:59.428714 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.428684 2585 patch_prober.go:28] interesting pod/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:57:59.428834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.428763 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:57:59.555670 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.555642 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp"] Jul 23 00:57:59.556086 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.556074 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9330c53f-9a5d-41d1-893c-6f9b0b70233f" containerName="revoke-keys" Jul 23 00:57:59.556132 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.556088 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9330c53f-9a5d-41d1-893c-6f9b0b70233f" containerName="revoke-keys" Jul 23 00:57:59.556166 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.556154 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9330c53f-9a5d-41d1-893c-6f9b0b70233f" containerName="revoke-keys" Jul 23 00:57:59.558458 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.558443 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:57:59.560428 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.560409 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:57:59.560558 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.560526 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 00:57:59.566445 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.566420 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp"] Jul 23 00:57:59.657992 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.657965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc916dc6-9a88-4895-8b1c-92597a5d4d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp\" (UID: \"cc916dc6-9a88-4895-8b1c-92597a5d4d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:57:59.758688 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.758656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc916dc6-9a88-4895-8b1c-92597a5d4d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp\" (UID: \"cc916dc6-9a88-4895-8b1c-92597a5d4d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:57:59.759304 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.759278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc916dc6-9a88-4895-8b1c-92597a5d4d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp\" (UID: \"cc916dc6-9a88-4895-8b1c-92597a5d4d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:57:59.870344 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:57:59.870315 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:58:00.012251 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:00.012219 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp"] Jul 23 00:58:00.012694 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:58:00.012662 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc916dc6_9a88_4895_8b1c_92597a5d4d09.slice/crio-03268fd0baae859cf79c33646a700cc277a0dae26ac95c8bf3d16ff42a419102 WatchSource:0}: Error finding container 03268fd0baae859cf79c33646a700cc277a0dae26ac95c8bf3d16ff42a419102: Status 404 returned error can't find the container with id 03268fd0baae859cf79c33646a700cc277a0dae26ac95c8bf3d16ff42a419102 Jul 23 00:58:00.153763 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:00.153734 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" event={"ID":"cc916dc6-9a88-4895-8b1c-92597a5d4d09","Type":"ContainerStarted","Data":"03268fd0baae859cf79c33646a700cc277a0dae26ac95c8bf3d16ff42a419102"} Jul 23 00:58:00.229666 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:00.229636 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:00.588937 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:00.588839 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:01.160394 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:01.160359 2585 generic.go:356] "Generic (PLEG): container finished" podID="cc916dc6-9a88-4895-8b1c-92597a5d4d09" containerID="c761ff658bb95205890cdfced391f6d67a8b5d9f5b6e752192630058f6c72cfe" exitCode=0 Jul 23 00:58:01.160789 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:01.160446 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" event={"ID":"cc916dc6-9a88-4895-8b1c-92597a5d4d09","Type":"ContainerDied","Data":"c761ff658bb95205890cdfced391f6d67a8b5d9f5b6e752192630058f6c72cfe"} Jul 23 00:58:02.298826 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:02.298806 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:58:02.382185 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:02.382154 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc916dc6-9a88-4895-8b1c-92597a5d4d09-maas-api-service-ca\") pod \"cc916dc6-9a88-4895-8b1c-92597a5d4d09\" (UID: \"cc916dc6-9a88-4895-8b1c-92597a5d4d09\") " Jul 23 00:58:02.382526 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:02.382504 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc916dc6-9a88-4895-8b1c-92597a5d4d09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc916dc6-9a88-4895-8b1c-92597a5d4d09" (UID: "cc916dc6-9a88-4895-8b1c-92597a5d4d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:58:02.483154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:02.483129 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc916dc6-9a88-4895-8b1c-92597a5d4d09-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:03.172287 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:03.172260 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" Jul 23 00:58:03.172505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:03.172302 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp" event={"ID":"cc916dc6-9a88-4895-8b1c-92597a5d4d09","Type":"ContainerDied","Data":"03268fd0baae859cf79c33646a700cc277a0dae26ac95c8bf3d16ff42a419102"} Jul 23 00:58:03.172505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:03.172357 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03268fd0baae859cf79c33646a700cc277a0dae26ac95c8bf3d16ff42a419102" Jul 23 00:58:04.428366 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.428346 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:58:04.503653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503621 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-data\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.503822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503687 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-certs\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.503822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503737 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-credential-socket\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.503822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503766 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-envoy\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.503822 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503800 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-istio-token\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.504056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503844 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08a865-b615-4d5f-9cba-a269574320bf-istio-podinfo\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.504056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503906 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-socket\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.504056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503952 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zshz\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-kube-api-access-4zshz\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.504056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.503981 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08a865-b615-4d5f-9cba-a269574320bf-istiod-ca-cert\") pod \"ee08a865-b615-4d5f-9cba-a269574320bf\" (UID: \"ee08a865-b615-4d5f-9cba-a269574320bf\") " Jul 23 00:58:04.504056 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504027 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:04.504314 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504217 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:04.504375 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504332 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.504375 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504347 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.504497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504424 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:04.504877 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504583 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:04.504877 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.504686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee08a865-b615-4d5f-9cba-a269574320bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:58:04.506842 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.506814 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:04.506989 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.506970 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:04.507109 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.507092 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-kube-api-access-4zshz" (OuterVolumeSpecName: "kube-api-access-4zshz") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "kube-api-access-4zshz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:04.507208 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.507187 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee08a865-b615-4d5f-9cba-a269574320bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee08a865-b615-4d5f-9cba-a269574320bf" (UID: "ee08a865-b615-4d5f-9cba-a269574320bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:58:04.605303 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605277 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.605303 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605301 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.605497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605313 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.605497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605324 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08a865-b615-4d5f-9cba-a269574320bf-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.605497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605336 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zshz\" (UniqueName: \"kubernetes.io/projected/ee08a865-b615-4d5f-9cba-a269574320bf-kube-api-access-4zshz\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.605497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605347 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08a865-b615-4d5f-9cba-a269574320bf-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:04.605497 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:04.605359 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08a865-b615-4d5f-9cba-a269574320bf-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:05.186659 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.186624 2585 generic.go:356] "Generic (PLEG): container finished" podID="ee08a865-b615-4d5f-9cba-a269574320bf" containerID="0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5" exitCode=0 Jul 23 00:58:05.186855 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.186675 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" Jul 23 00:58:05.186855 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.186655 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" event={"ID":"ee08a865-b615-4d5f-9cba-a269574320bf","Type":"ContainerDied","Data":"0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5"} Jul 23 00:58:05.186855 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.186806 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77" event={"ID":"ee08a865-b615-4d5f-9cba-a269574320bf","Type":"ContainerDied","Data":"ba74f864516a33815fad4314c342e7e35c4b3495d4f0d9503c599602032a9392"} Jul 23 00:58:05.186855 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.186838 2585 scope.go:117] "RemoveContainer" containerID="0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5" Jul 23 00:58:05.197074 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.197058 2585 scope.go:117] "RemoveContainer" containerID="270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8" Jul 23 00:58:05.205238 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.205217 2585 scope.go:117] "RemoveContainer" containerID="0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5" Jul 23 00:58:05.206158 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:58:05.205951 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5\": container with ID starting with 0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5 not found: ID does not exist" containerID="0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5" Jul 23 00:58:05.206158 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.206015 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5"} err="failed to get container status \"0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5\": rpc error: code = NotFound desc = could not find container \"0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5\": container with ID starting with 0550238ab33420dc97e986801ec6c1a253c5d9ef6df1ccdcfc28c884aff61ff5 not found: ID does not exist" Jul 23 00:58:05.206158 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.206039 2585 scope.go:117] "RemoveContainer" containerID="270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8" Jul 23 00:58:05.206790 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:58:05.206765 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8\": container with ID starting with 270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8 not found: ID does not exist" containerID="270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8" Jul 23 00:58:05.206891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.206796 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8"} err="failed to get container status \"270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8\": rpc error: code = NotFound desc = could not find container \"270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8\": container with ID starting with 270e41b10366ca96a70cefb1f2d6604e31d1a657e2df9020e968f6484ff409c8 not found: ID does not exist" Jul 23 00:58:05.210760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.210739 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77"] Jul 23 00:58:05.212472 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.212437 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-23cf83-openshift-default-5c468d595c-lfs77"] Jul 23 00:58:05.412606 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:05.412575 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" path="/var/lib/kubelet/pods/ee08a865-b615-4d5f-9cba-a269574320bf/volumes" Jul 23 00:58:15.007907 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:15.007875 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:19.652449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:19.652415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:19.663811 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:19.663779 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp"] Jul 23 00:58:19.666298 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:19.666275 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fe2f-9j2tp"] Jul 23 00:58:20.062611 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:20.062577 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:21.411952 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:21.411917 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc916dc6-9a88-4895-8b1c-92597a5d4d09" path="/var/lib/kubelet/pods/cc916dc6-9a88-4895-8b1c-92597a5d4d09/volumes" Jul 23 00:58:25.654228 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:25.654138 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:25.754834 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:25.754797 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:25.915570 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:25.915439 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n"] Jul 23 00:58:25.915754 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:25.915725 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" containerID="cri-o://52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114" gracePeriod=30 Jul 23 00:58:25.939652 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:25.939623 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:26.183893 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.183806 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:26.561760 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.561728 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:26.896066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.895984 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm"] Jul 23 00:58:26.896415 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896397 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" Jul 23 00:58:26.896415 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896409 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" Jul 23 00:58:26.896515 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896435 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc916dc6-9a88-4895-8b1c-92597a5d4d09" containerName="revoke-keys" Jul 23 00:58:26.896515 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896441 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc916dc6-9a88-4895-8b1c-92597a5d4d09" containerName="revoke-keys" Jul 23 00:58:26.896586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896515 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" Jul 23 00:58:26.896586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896524 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" Jul 23 00:58:26.896586 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.896533 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc916dc6-9a88-4895-8b1c-92597a5d4d09" containerName="revoke-keys" Jul 23 00:58:26.899803 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.899783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:26.901747 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.901722 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b90c57-openshift-default-dockercfg-vz9xm\"" Jul 23 00:58:26.913341 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.913319 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm"] Jul 23 00:58:26.928307 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:26.928279 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:27.007621 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-socket\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007823 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-certs\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007823 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-token\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007823 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007717 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-data\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007826 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-credential-socket\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007865 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-podinfo\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f5z\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-kube-api-access-49f5z\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.007986 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.007946 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-envoy\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.008184 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.008048 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a5e059-9e4b-4476-884f-f6b3437728ae-istiod-ca-cert\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109181 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109138 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a5e059-9e4b-4476-884f-f6b3437728ae-istiod-ca-cert\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109197 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-socket\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109222 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-certs\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109363 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-token\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109579 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109367 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-data\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109579 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-credential-socket\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109579 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-podinfo\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109579 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109517 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f5z\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-kube-api-access-49f5z\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109803 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-envoy\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-data\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109961 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a5e059-9e4b-4476-884f-f6b3437728ae-istiod-ca-cert\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.109980 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109971 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-certs\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.110191 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.109972 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-credential-socket\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.110191 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.110152 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-socket\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.112431 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.112409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-envoy\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.112587 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.112568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-podinfo\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.116612 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.116589 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-token\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.116711 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.116690 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f5z\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-kube-api-access-49f5z\") pod \"e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.213855 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.213774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:27.369704 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.369680 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm"] Jul 23 00:58:27.371201 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:58:27.371179 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a5e059_9e4b_4476_884f_f6b3437728ae.slice/crio-ff2f33cf5748100f4ddca1c5d7ece508388d69178cd034a0ff6f0d3246d8d617 WatchSource:0}: Error finding container ff2f33cf5748100f4ddca1c5d7ece508388d69178cd034a0ff6f0d3246d8d617: Status 404 returned error can't find the container with id ff2f33cf5748100f4ddca1c5d7ece508388d69178cd034a0ff6f0d3246d8d617 Jul 23 00:58:27.373446 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.373417 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:27.373552 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.373495 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:27.373552 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.373533 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:27.391017 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:27.390982 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:28.312382 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:28.312345 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" event={"ID":"65a5e059-9e4b-4476-884f-f6b3437728ae","Type":"ContainerStarted","Data":"3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de"} Jul 23 00:58:28.312805 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:28.312429 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" event={"ID":"65a5e059-9e4b-4476-884f-f6b3437728ae","Type":"ContainerStarted","Data":"ff2f33cf5748100f4ddca1c5d7ece508388d69178cd034a0ff6f0d3246d8d617"} Jul 23 00:58:28.332389 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:28.332342 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podStartSLOduration=2.332326587 podStartE2EDuration="2.332326587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:58:28.330814529 +0000 UTC m=+2337.627509490" watchObservedRunningTime="2026-07-23 00:58:28.332326587 +0000 UTC m=+2337.629021616" Jul 23 00:58:29.214080 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:29.214045 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:29.219034 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:29.219007 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:29.317616 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:29.317588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:29.318385 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:29.318368 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:30.417266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:30.417230 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:30.509349 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:30.509313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:30.572361 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:30.572330 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:30.705680 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:30.705599 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:31.025983 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.025952 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:31.126410 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.126379 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:31.166006 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.165981 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:58:31.248993 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.248968 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-podinfo\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249178 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249001 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-certs\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249178 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249022 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-data\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249178 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249064 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-credential-socket\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249178 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249091 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-envoy\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249178 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249126 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-token\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249178 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249157 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91cb4da9-1be6-446b-8922-c3cfa021cb72-istiod-ca-cert\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249193 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5g5r\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-kube-api-access-k5g5r\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249261 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-socket\") pod \"91cb4da9-1be6-446b-8922-c3cfa021cb72\" (UID: \"91cb4da9-1be6-446b-8922-c3cfa021cb72\") " Jul 23 00:58:31.249545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249268 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:31.249545 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:31.249762 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249571 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.249762 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249618 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:31.249762 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.249648 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cb4da9-1be6-446b-8922-c3cfa021cb72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:58:31.250158 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.250133 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-data" (OuterVolumeSpecName: "istio-data") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:31.251176 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.251155 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:58:31.251304 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.251285 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-kube-api-access-k5g5r" (OuterVolumeSpecName: "kube-api-access-k5g5r") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "kube-api-access-k5g5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:31.251414 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.251397 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:31.251610 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.251594 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-token" (OuterVolumeSpecName: "istio-token") pod "91cb4da9-1be6-446b-8922-c3cfa021cb72" (UID: "91cb4da9-1be6-446b-8922-c3cfa021cb72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:31.329493 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.329400 2585 generic.go:356] "Generic (PLEG): container finished" podID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerID="52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114" exitCode=0 Jul 23 00:58:31.329493 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.329446 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" Jul 23 00:58:31.329646 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.329501 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" event={"ID":"91cb4da9-1be6-446b-8922-c3cfa021cb72","Type":"ContainerDied","Data":"52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114"} Jul 23 00:58:31.329646 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.329558 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n" event={"ID":"91cb4da9-1be6-446b-8922-c3cfa021cb72","Type":"ContainerDied","Data":"66de57802d8e7c2a9ddbc2f2858823510b6735169b03e8cae89dbba478d921a0"} Jul 23 00:58:31.329646 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.329589 2585 scope.go:117] "RemoveContainer" containerID="52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114" Jul 23 00:58:31.339663 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.339647 2585 scope.go:117] "RemoveContainer" containerID="1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a" Jul 23 00:58:31.348284 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.348265 2585 scope.go:117] "RemoveContainer" containerID="52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114" Jul 23 00:58:31.348556 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:58:31.348527 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114\": container with ID starting with 52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114 not found: ID does not exist" containerID="52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114" Jul 23 00:58:31.348656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.348567 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114"} err="failed to get container status \"52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114\": rpc error: code = NotFound desc = could not find container \"52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114\": container with ID starting with 52ddcbd91a4529ade0080b559abb56b167579fb9144b0ef39413981d00aae114 not found: ID does not exist" Jul 23 00:58:31.348656 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.348593 2585 scope.go:117] "RemoveContainer" containerID="1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a" Jul 23 00:58:31.348865 ip-10-0-139-142 kubenswrapper[2585]: E0723 00:58:31.348848 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a\": container with ID starting with 1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a not found: ID does not exist" containerID="1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a" Jul 23 00:58:31.348909 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.348871 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a"} err="failed to get container status \"1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a\": rpc error: code = NotFound desc = could not find container \"1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a\": container with ID starting with 1ac8f420e1c95a1e2142aff1c59df3d835d0b251ca758f55f35b58c69ce5bf7a not found: ID does not exist" Jul 23 00:58:31.349899 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.349878 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n"] Jul 23 00:58:31.350228 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350216 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350231 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350240 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350248 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350257 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350266 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350265 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350412 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350273 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91cb4da9-1be6-446b-8922-c3cfa021cb72-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.350412 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.350281 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5g5r\" (UniqueName: \"kubernetes.io/projected/91cb4da9-1be6-446b-8922-c3cfa021cb72-kube-api-access-k5g5r\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 00:58:31.353695 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.353673 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e6fe2f-openshift-default-674dcbbcd5-7l47n"] Jul 23 00:58:31.412349 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.412322 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" path="/var/lib/kubelet/pods/91cb4da9-1be6-446b-8922-c3cfa021cb72/volumes" Jul 23 00:58:31.550010 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:31.549980 2585 scope.go:117] "RemoveContainer" containerID="4eefc34c52d4d90b14d8ae4d456104489434de9784b6d57b5a8913ea1c4e97f9" Jul 23 00:58:34.871743 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:34.871707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:35.225297 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.225208 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:35.522542 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.522510 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s"] Jul 23 00:58:35.523048 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523022 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" Jul 23 00:58:35.523048 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523049 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" Jul 23 00:58:35.523215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523070 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" Jul 23 00:58:35.523215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523077 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" Jul 23 00:58:35.523215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523092 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" Jul 23 00:58:35.523215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523097 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee08a865-b615-4d5f-9cba-a269574320bf" containerName="istio-proxy" Jul 23 00:58:35.523215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523165 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" Jul 23 00:58:35.523215 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.523172 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cb4da9-1be6-446b-8922-c3cfa021cb72" containerName="istio-proxy" Jul 23 00:58:35.528125 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.528107 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.530324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.530302 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-75b665-openshift-default-dockercfg-kzsv7\"" Jul 23 00:58:35.538410 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.538386 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s"] Jul 23 00:58:35.547795 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.547751 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:35.690203 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-certs\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ns8\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-kube-api-access-d6ns8\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690291 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istiod-ca-cert\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690378 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690353 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-socket\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-credential-socket\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-data\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690505 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-podinfo\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-envoy\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.690622 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.690575 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-token\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791730 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-socket\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791730 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-credential-socket\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791730 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791714 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-data\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-podinfo\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-envoy\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791794 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-token\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791839 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-certs\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791894 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ns8\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-kube-api-access-d6ns8\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.791994 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.791928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istiod-ca-cert\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.792301 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.792104 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-credential-socket\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.792301 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.792170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-socket\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.792301 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.792190 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-data\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.792524 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.792503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-certs\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.792665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.792643 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istiod-ca-cert\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.794133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.794111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-envoy\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.794324 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.794305 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-podinfo\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.804436 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.804412 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-token\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.804759 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.804738 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ns8\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-kube-api-access-d6ns8\") pod \"e2e-shared-b-75b665-openshift-default-54c868597f-62r6s\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.821653 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.821629 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:35.840665 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.840643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:35.979391 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.979367 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s"] Jul 23 00:58:35.981457 ip-10-0-139-142 kubenswrapper[2585]: W0723 00:58:35.981430 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad146f8_43b8_4c0b_87b7_7db166f5ac2e.slice/crio-c0c2d8175e6ab9a8b87ae0369cbefa75d155a77d2cfd4e0e97a479746b83b140 WatchSource:0}: Error finding container c0c2d8175e6ab9a8b87ae0369cbefa75d155a77d2cfd4e0e97a479746b83b140: Status 404 returned error can't find the container with id c0c2d8175e6ab9a8b87ae0369cbefa75d155a77d2cfd4e0e97a479746b83b140 Jul 23 00:58:35.983789 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.983761 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:35.983862 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.983823 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:35.983862 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:35.983850 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:36.360907 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:36.360807 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" event={"ID":"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e","Type":"ContainerStarted","Data":"d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9"} Jul 23 00:58:36.360907 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:36.360869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" event={"ID":"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e","Type":"ContainerStarted","Data":"c0c2d8175e6ab9a8b87ae0369cbefa75d155a77d2cfd4e0e97a479746b83b140"} Jul 23 00:58:36.399004 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:36.398958 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podStartSLOduration=1.398942257 podStartE2EDuration="1.398942257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:58:36.397119544 +0000 UTC m=+2345.693814515" watchObservedRunningTime="2026-07-23 00:58:36.398942257 +0000 UTC m=+2345.695637225" Jul 23 00:58:36.841283 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:36.841251 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:37.846107 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:37.846078 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:38.366924 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:38.366886 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:38.378053 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:38.378025 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:38.379235 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:38.379205 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:38.461135 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:38.461096 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:38.533645 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:38.533600 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:38.673300 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:38.673223 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:39.039142 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:39.039108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:39.140111 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:39.140076 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:42.908218 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:42.908178 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:43.047233 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:43.047201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:43.148280 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:43.148249 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:45.359652 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:45.359607 2585 patch_prober.go:28] interesting pod/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 00:58:45.360072 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:45.359703 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 00:58:46.425699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:46.425614 2585 generic.go:356] "Generic (PLEG): container finished" podID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerID="3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de" exitCode=0 Jul 23 00:58:46.425699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:46.425683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" event={"ID":"65a5e059-9e4b-4476-884f-f6b3437728ae","Type":"ContainerDied","Data":"3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de"} Jul 23 00:58:46.426130 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:46.426078 2585 scope.go:117] "RemoveContainer" containerID="3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de" Jul 23 00:58:46.427099 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:46.427067 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:46.427195 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:46.427152 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:46.427195 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:46.427186 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:47.441066 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:47.441030 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" event={"ID":"65a5e059-9e4b-4476-884f-f6b3437728ae","Type":"ContainerStarted","Data":"ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99"} Jul 23 00:58:48.215179 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:48.215145 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:48.215574 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:48.215482 2585 patch_prober.go:28] interesting pod/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 23 00:58:48.215751 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:48.215731 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 23 00:58:49.214814 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:49.214776 2585 patch_prober.go:28] interesting pod/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 23 00:58:49.215420 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:49.215387 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 23 00:58:50.214690 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:50.214646 2585 patch_prober.go:28] interesting pod/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 23 00:58:50.215001 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:50.214968 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 23 00:58:51.214675 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:51.214628 2585 patch_prober.go:28] interesting pod/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 23 00:58:51.214891 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:51.214700 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 23 00:58:51.902513 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:51.902413 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:52.004174 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:52.004139 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:52.218695 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:52.218620 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:52.219154 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:52.219131 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:52.219790 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:52.219771 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 00:58:54.419639 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.419602 2585 patch_prober.go:28] interesting pod/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 00:58:54.421911 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.419684 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 00:58:54.491528 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.491504 2585 generic.go:356] "Generic (PLEG): container finished" podID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerID="d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9" exitCode=0 Jul 23 00:58:54.491628 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.491572 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" event={"ID":"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e","Type":"ContainerDied","Data":"d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9"} Jul 23 00:58:54.491946 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.491932 2585 scope.go:117] "RemoveContainer" containerID="d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9" Jul 23 00:58:54.492870 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.492843 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:54.492929 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.492907 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:54.492966 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:54.492936 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:58:55.511792 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:55.511746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" event={"ID":"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e","Type":"ContainerStarted","Data":"d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f"} Jul 23 00:58:55.841163 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:55.841073 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:55.841353 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:55.841314 2585 patch_prober.go:28] interesting pod/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 23 00:58:55.841399 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:55.841367 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 23 00:58:56.841449 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:56.841400 2585 patch_prober.go:28] interesting pod/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 23 00:58:56.841846 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:56.841514 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 23 00:58:57.841165 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:57.841127 2585 patch_prober.go:28] interesting pod/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 23 00:58:57.841347 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:57.841197 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 23 00:58:58.841744 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:58.841710 2585 patch_prober.go:28] interesting pod/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 23 00:58:58.842133 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:58.841773 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 23 00:58:59.793900 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:59.793870 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:59.844362 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:59.844329 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:59.844775 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:59.844664 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:59.845270 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:59.845250 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 00:58:59.909699 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:58:59.909660 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:00.004969 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:59:00.004935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:25.485094 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:59:25.485052 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:31.347626 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:59:31.347594 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 00:59:31.351773 ip-10-0-139-142 kubenswrapper[2585]: I0723 00:59:31.351748 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 01:00:00.146121 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.146081 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh"] Jul 23 01:00:00.149961 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.149939 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" Jul 23 01:00:00.150201 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.150173 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b"] Jul 23 01:00:00.152128 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.152108 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 01:00:00.153938 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.153920 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q"] Jul 23 01:00:00.154038 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.154027 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" Jul 23 01:00:00.157589 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.157403 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" Jul 23 01:00:00.159400 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.159368 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh"] Jul 23 01:00:00.162094 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.162077 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b"] Jul 23 01:00:00.163270 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.163234 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" Jul 23 01:00:00.165309 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.165286 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q"] Jul 23 01:00:00.176123 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.176102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" Jul 23 01:00:00.217041 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.217017 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" Jul 23 01:00:00.389947 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.389922 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q"] Jul 23 01:00:00.391537 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:00:00.391508 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e26d2d_afce_42c8_8d76_87b58ac398a1.slice/crio-705a641a33047378ade50487fdfcefa7eddb207a8ddaa7ce43fa0896662c7faf WatchSource:0}: Error finding container 705a641a33047378ade50487fdfcefa7eddb207a8ddaa7ce43fa0896662c7faf: Status 404 returned error can't find the container with id 705a641a33047378ade50487fdfcefa7eddb207a8ddaa7ce43fa0896662c7faf Jul 23 01:00:00.538542 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.538514 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh"] Jul 23 01:00:00.572565 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.572543 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b"] Jul 23 01:00:00.575247 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:00:00.575224 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7094f88c_9d60_4fec_9c00_2d6f09b93035.slice/crio-97b118975fea2e659b8d1d0b40f8ab8ea67df2eeb8a4544b269d516107b18b82 WatchSource:0}: Error finding container 97b118975fea2e659b8d1d0b40f8ab8ea67df2eeb8a4544b269d516107b18b82: Status 404 returned error can't find the container with id 97b118975fea2e659b8d1d0b40f8ab8ea67df2eeb8a4544b269d516107b18b82 Jul 23 01:00:00.891339 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.891306 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" event={"ID":"120ef1c7-ca0e-43ab-adad-2be678e90359","Type":"ContainerStarted","Data":"5e86a069aef811868899a39cebd11868c79db409e826bf5b017a40ab72e02c01"} Jul 23 01:00:00.891528 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.891369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" event={"ID":"120ef1c7-ca0e-43ab-adad-2be678e90359","Type":"ContainerStarted","Data":"c7b49ab92ef327ed8ee188410f2fb649f3857ebf9f54b76bcae7b1af2493a3fd"} Jul 23 01:00:00.892860 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.892836 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" event={"ID":"42e26d2d-afce-42c8-8d76-87b58ac398a1","Type":"ContainerStarted","Data":"ba4502ff475c4c79e43634ee9ec00989197335d45d29ed314adf6d92b1a0b8e4"} Jul 23 01:00:00.892970 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.892882 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" event={"ID":"42e26d2d-afce-42c8-8d76-87b58ac398a1","Type":"ContainerStarted","Data":"705a641a33047378ade50487fdfcefa7eddb207a8ddaa7ce43fa0896662c7faf"} Jul 23 01:00:00.893977 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.893956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" event={"ID":"7094f88c-9d60-4fec-9c00-2d6f09b93035","Type":"ContainerStarted","Data":"97b118975fea2e659b8d1d0b40f8ab8ea67df2eeb8a4544b269d516107b18b82"} Jul 23 01:00:00.906095 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.906056 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" podStartSLOduration=0.906042886 podStartE2EDuration="906.042886ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:00.903541427 +0000 UTC m=+2430.200236422" watchObservedRunningTime="2026-07-23 01:00:00.906042886 +0000 UTC m=+2430.202737854" Jul 23 01:00:00.917561 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:00.917523 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" podStartSLOduration=0.91750975 podStartE2EDuration="917.50975ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:00.915805537 +0000 UTC m=+2430.212500524" watchObservedRunningTime="2026-07-23 01:00:00.91750975 +0000 UTC m=+2430.214204717" Jul 23 01:00:01.900853 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:01.900723 2585 generic.go:356] "Generic (PLEG): container finished" podID="7094f88c-9d60-4fec-9c00-2d6f09b93035" containerID="76d74d49e06246425c46ce456bb6e4004764831cf4c7c38a7349f064442c7b57" exitCode=0 Jul 23 01:00:01.900853 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:01.900821 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" event={"ID":"7094f88c-9d60-4fec-9c00-2d6f09b93035","Type":"ContainerDied","Data":"76d74d49e06246425c46ce456bb6e4004764831cf4c7c38a7349f064442c7b57"} Jul 23 01:00:01.902677 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:01.902649 2585 generic.go:356] "Generic (PLEG): container finished" podID="120ef1c7-ca0e-43ab-adad-2be678e90359" containerID="5e86a069aef811868899a39cebd11868c79db409e826bf5b017a40ab72e02c01" exitCode=0 Jul 23 01:00:01.902823 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:01.902727 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" event={"ID":"120ef1c7-ca0e-43ab-adad-2be678e90359","Type":"ContainerDied","Data":"5e86a069aef811868899a39cebd11868c79db409e826bf5b017a40ab72e02c01"} Jul 23 01:00:01.904533 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:01.904508 2585 generic.go:356] "Generic (PLEG): container finished" podID="42e26d2d-afce-42c8-8d76-87b58ac398a1" containerID="ba4502ff475c4c79e43634ee9ec00989197335d45d29ed314adf6d92b1a0b8e4" exitCode=0 Jul 23 01:00:01.904647 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:01.904535 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" event={"ID":"42e26d2d-afce-42c8-8d76-87b58ac398a1","Type":"ContainerDied","Data":"ba4502ff475c4c79e43634ee9ec00989197335d45d29ed314adf6d92b1a0b8e4"} Jul 23 01:00:03.089080 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.089059 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" Jul 23 01:00:03.092764 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.092745 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" Jul 23 01:00:03.124985 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.124954 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" Jul 23 01:00:03.919000 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.918970 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" Jul 23 01:00:03.919000 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.918981 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-xp7rh" event={"ID":"120ef1c7-ca0e-43ab-adad-2be678e90359","Type":"ContainerDied","Data":"c7b49ab92ef327ed8ee188410f2fb649f3857ebf9f54b76bcae7b1af2493a3fd"} Jul 23 01:00:03.919225 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.919031 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7b49ab92ef327ed8ee188410f2fb649f3857ebf9f54b76bcae7b1af2493a3fd" Jul 23 01:00:03.920444 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.920424 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" Jul 23 01:00:03.920576 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.920451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q" event={"ID":"42e26d2d-afce-42c8-8d76-87b58ac398a1","Type":"ContainerDied","Data":"705a641a33047378ade50487fdfcefa7eddb207a8ddaa7ce43fa0896662c7faf"} Jul 23 01:00:03.920576 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.920514 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705a641a33047378ade50487fdfcefa7eddb207a8ddaa7ce43fa0896662c7faf" Jul 23 01:00:03.922207 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.922182 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" event={"ID":"7094f88c-9d60-4fec-9c00-2d6f09b93035","Type":"ContainerDied","Data":"97b118975fea2e659b8d1d0b40f8ab8ea67df2eeb8a4544b269d516107b18b82"} Jul 23 01:00:03.922207 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.922205 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b" Jul 23 01:00:03.922358 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:00:03.922217 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b118975fea2e659b8d1d0b40f8ab8ea67df2eeb8a4544b269d516107b18b82" Jul 23 01:01:26.884778 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:01:26.884697 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:26.968644 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:01:26.968612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:32.344689 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:01:32.344659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:33.186340 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:01:33.186311 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:38.781287 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:01:38.781255 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:05.770716 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:05.770682 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:06.100232 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:06.100150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:19.252743 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:19.252709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:22.156910 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:22.156877 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:31.603246 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:31.603210 2585 scope.go:117] "RemoveContainer" containerID="96b7dd94705f19c8c1a63374248f4aeaaba1619938a4af659d19af3c00a24f1c" Jul 23 01:02:52.683064 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:52.683031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:53.364535 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:02:53.364499 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:03.883628 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:03.883596 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:12.848675 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:12.848628 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:31.626143 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:31.626111 2585 scope.go:117] "RemoveContainer" containerID="c285df8cf8badf0a9e52780859f20c899701b329758055d4c74cb13b7cd4ecfe" Jul 23 01:03:43.755091 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:43.755052 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:43.856641 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:43.856607 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:52.175751 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:52.175717 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:52.641655 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:52.641620 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:52.741726 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:03:52.741687 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:00.395888 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:00.395852 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:00.644711 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:00.644672 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:00.746343 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:00.746315 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:31.385173 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:31.385082 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 01:04:31.391122 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:31.391098 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 01:04:31.647849 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:31.647765 2585 scope.go:117] "RemoveContainer" containerID="c761ff658bb95205890cdfced391f6d67a8b5d9f5b6e752192630058f6c72cfe" Jul 23 01:04:31.656563 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:31.656543 2585 scope.go:117] "RemoveContainer" containerID="4ab477183e28aaeea079e795849b06cf7e2827b13d767a7d3ac8a1a5a22538a6" Jul 23 01:04:33.794851 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:33.794811 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:33.892064 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:33.892028 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:57.268989 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:57.268955 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:58.200734 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:58.200639 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:59.246611 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.246577 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7655-140249961893168"] Jul 23 01:04:59.246991 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.246977 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120ef1c7-ca0e-43ab-adad-2be678e90359" containerName="cleanup" Jul 23 01:04:59.246991 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.246989 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="120ef1c7-ca0e-43ab-adad-2be678e90359" containerName="cleanup" Jul 23 01:04:59.247066 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.246997 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7094f88c-9d60-4fec-9c00-2d6f09b93035" containerName="cleanup" Jul 23 01:04:59.247066 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.247002 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7094f88c-9d60-4fec-9c00-2d6f09b93035" containerName="cleanup" Jul 23 01:04:59.247066 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.247010 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e26d2d-afce-42c8-8d76-87b58ac398a1" containerName="cleanup" Jul 23 01:04:59.247066 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.247015 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e26d2d-afce-42c8-8d76-87b58ac398a1" containerName="cleanup" Jul 23 01:04:59.247193 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.247072 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e26d2d-afce-42c8-8d76-87b58ac398a1" containerName="cleanup" Jul 23 01:04:59.247193 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.247082 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7094f88c-9d60-4fec-9c00-2d6f09b93035" containerName="cleanup" Jul 23 01:04:59.247193 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.247089 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="120ef1c7-ca0e-43ab-adad-2be678e90359" containerName="cleanup" Jul 23 01:04:59.250221 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.250204 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:04:59.252430 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.252412 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xnsh4\"" Jul 23 01:04:59.257251 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.257228 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961893168"] Jul 23 01:04:59.320780 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.320739 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d24k\" (UniqueName: \"kubernetes.io/projected/25996520-64b1-4291-ab92-54723851acf0-kube-api-access-9d24k\") pod \"test-curl-7655-140249961893168\" (UID: \"25996520-64b1-4291-ab92-54723851acf0\") " pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:04:59.422038 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.422000 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d24k\" (UniqueName: \"kubernetes.io/projected/25996520-64b1-4291-ab92-54723851acf0-kube-api-access-9d24k\") pod \"test-curl-7655-140249961893168\" (UID: \"25996520-64b1-4291-ab92-54723851acf0\") " pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:04:59.429437 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.429407 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d24k\" (UniqueName: \"kubernetes.io/projected/25996520-64b1-4291-ab92-54723851acf0-kube-api-access-9d24k\") pod \"test-curl-7655-140249961893168\" (UID: \"25996520-64b1-4291-ab92-54723851acf0\") " pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:04:59.560831 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.560744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:04:59.697124 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.697098 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961893168"] Jul 23 01:04:59.698982 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:04:59.698955 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25996520_64b1_4291_ab92_54723851acf0.slice/crio-32db80e15a3bb11aa5a121db253a4f4e8821d7ba353d6736920db957a4d9ba3f WatchSource:0}: Error finding container 32db80e15a3bb11aa5a121db253a4f4e8821d7ba353d6736920db957a4d9ba3f: Status 404 returned error can't find the container with id 32db80e15a3bb11aa5a121db253a4f4e8821d7ba353d6736920db957a4d9ba3f Jul 23 01:04:59.700354 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:04:59.700336 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:05:00.566533 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:00.566490 2585 generic.go:356] "Generic (PLEG): container finished" podID="25996520-64b1-4291-ab92-54723851acf0" containerID="8ecb316ef428741a6951f4584650218e555770baca7f4a0b1b0d5de8b3e9fdce" exitCode=0 Jul 23 01:05:00.566966 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:00.566543 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961893168" event={"ID":"25996520-64b1-4291-ab92-54723851acf0","Type":"ContainerDied","Data":"8ecb316ef428741a6951f4584650218e555770baca7f4a0b1b0d5de8b3e9fdce"} Jul 23 01:05:00.566966 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:00.566572 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961893168" event={"ID":"25996520-64b1-4291-ab92-54723851acf0","Type":"ContainerStarted","Data":"32db80e15a3bb11aa5a121db253a4f4e8821d7ba353d6736920db957a4d9ba3f"} Jul 23 01:05:01.718753 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.718724 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:05:01.743178 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.743144 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7655-140249961893168_25996520-64b1-4291-ab92-54723851acf0/test-curl-7655-140249961893168/0.log" Jul 23 01:05:01.751346 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.751321 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d24k\" (UniqueName: \"kubernetes.io/projected/25996520-64b1-4291-ab92-54723851acf0-kube-api-access-9d24k\") pod \"25996520-64b1-4291-ab92-54723851acf0\" (UID: \"25996520-64b1-4291-ab92-54723851acf0\") " Jul 23 01:05:01.753648 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.753621 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25996520-64b1-4291-ab92-54723851acf0-kube-api-access-9d24k" (OuterVolumeSpecName: "kube-api-access-9d24k") pod "25996520-64b1-4291-ab92-54723851acf0" (UID: "25996520-64b1-4291-ab92-54723851acf0"). InnerVolumeSpecName "kube-api-access-9d24k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:01.769616 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.769585 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7655-140249961893168"] Jul 23 01:05:01.772555 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.772533 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7655-140249961893168"] Jul 23 01:05:01.852511 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.852418 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d24k\" (UniqueName: \"kubernetes.io/projected/25996520-64b1-4291-ab92-54723851acf0-kube-api-access-9d24k\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:01.902570 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.902535 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7655-140249961893424"] Jul 23 01:05:01.902973 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.902957 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25996520-64b1-4291-ab92-54723851acf0" containerName="test-curl-7655-140249961893168" Jul 23 01:05:01.903020 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.902976 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="25996520-64b1-4291-ab92-54723851acf0" containerName="test-curl-7655-140249961893168" Jul 23 01:05:01.903099 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.903088 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="25996520-64b1-4291-ab92-54723851acf0" containerName="test-curl-7655-140249961893168" Jul 23 01:05:01.905999 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.905982 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:01.911702 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.911676 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961893424"] Jul 23 01:05:01.952855 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:01.952821 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7d6l\" (UniqueName: \"kubernetes.io/projected/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0-kube-api-access-f7d6l\") pod \"test-curl-7655-140249961893424\" (UID: \"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0\") " pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:02.053694 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.053655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7d6l\" (UniqueName: \"kubernetes.io/projected/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0-kube-api-access-f7d6l\") pod \"test-curl-7655-140249961893424\" (UID: \"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0\") " pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:02.062057 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.062028 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7d6l\" (UniqueName: \"kubernetes.io/projected/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0-kube-api-access-f7d6l\") pod \"test-curl-7655-140249961893424\" (UID: \"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0\") " pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:02.216191 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.216098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:02.351289 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.351262 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961893424"] Jul 23 01:05:02.353319 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:02.353293 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcfae845_a07d_4c6f_90ab_f9c62ceabcd0.slice/crio-a529cd55cbc74e7d369f539967522583651d7638c11dc67e89d5a13e4cc5e3de WatchSource:0}: Error finding container a529cd55cbc74e7d369f539967522583651d7638c11dc67e89d5a13e4cc5e3de: Status 404 returned error can't find the container with id a529cd55cbc74e7d369f539967522583651d7638c11dc67e89d5a13e4cc5e3de Jul 23 01:05:02.578872 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.578818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961893424" event={"ID":"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0","Type":"ContainerStarted","Data":"a529cd55cbc74e7d369f539967522583651d7638c11dc67e89d5a13e4cc5e3de"} Jul 23 01:05:02.580096 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.580079 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:05:02.580191 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:02.580083 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32db80e15a3bb11aa5a121db253a4f4e8821d7ba353d6736920db957a4d9ba3f" Jul 23 01:05:02.589965 ip-10-0-139-142 kubenswrapper[2585]: E0723 01:05:02.589934 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7655-140249961893168\" is forbidden: User \"system:node:ip-10-0-139-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-142.ec2.internal' and this object" podUID="25996520-64b1-4291-ab92-54723851acf0" pod="opendatahub/test-curl-7655-140249961893168" Jul 23 01:05:03.412337 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:03.412303 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25996520-64b1-4291-ab92-54723851acf0" path="/var/lib/kubelet/pods/25996520-64b1-4291-ab92-54723851acf0/volumes" Jul 23 01:05:03.586878 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:03.586846 2585 generic.go:356] "Generic (PLEG): container finished" podID="dcfae845-a07d-4c6f-90ab-f9c62ceabcd0" containerID="13abb1e2ba98e84772cc6d0f7a960fdb975f6d544f58947008f6cd0a8c0e6877" exitCode=0 Jul 23 01:05:03.587054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:03.586907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961893424" event={"ID":"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0","Type":"ContainerDied","Data":"13abb1e2ba98e84772cc6d0f7a960fdb975f6d544f58947008f6cd0a8c0e6877"} Jul 23 01:05:04.737838 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.737807 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:04.759484 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.759437 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7655-140249961893424_dcfae845-a07d-4c6f-90ab-f9c62ceabcd0/test-curl-7655-140249961893424/0.log" Jul 23 01:05:04.776892 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.776866 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7d6l\" (UniqueName: \"kubernetes.io/projected/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0-kube-api-access-f7d6l\") pod \"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0\" (UID: \"dcfae845-a07d-4c6f-90ab-f9c62ceabcd0\") " Jul 23 01:05:04.778901 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.778867 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0-kube-api-access-f7d6l" (OuterVolumeSpecName: "kube-api-access-f7d6l") pod "dcfae845-a07d-4c6f-90ab-f9c62ceabcd0" (UID: "dcfae845-a07d-4c6f-90ab-f9c62ceabcd0"). InnerVolumeSpecName "kube-api-access-f7d6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:04.784608 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.784583 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7655-140249961893424"] Jul 23 01:05:04.787903 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.787883 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7655-140249961893424"] Jul 23 01:05:04.877826 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.877740 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7d6l\" (UniqueName: \"kubernetes.io/projected/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0-kube-api-access-f7d6l\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:04.949689 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.949651 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:04.950098 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.950083 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfae845-a07d-4c6f-90ab-f9c62ceabcd0" containerName="test-curl-7655-140249961893424" Jul 23 01:05:04.950147 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.950101 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfae845-a07d-4c6f-90ab-f9c62ceabcd0" containerName="test-curl-7655-140249961893424" Jul 23 01:05:04.950183 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.950169 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcfae845-a07d-4c6f-90ab-f9c62ceabcd0" containerName="test-curl-7655-140249961893424" Jul 23 01:05:04.959554 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.959526 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:04.960944 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.960915 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:04.978775 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:04.978737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhb82\" (UniqueName: \"kubernetes.io/projected/a10485c9-afa9-4726-81d0-e7e45f1d91fd-kube-api-access-mhb82\") pod \"test-curl-7655-140249961882656\" (UID: \"a10485c9-afa9-4726-81d0-e7e45f1d91fd\") " pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:05.080194 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.080157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhb82\" (UniqueName: \"kubernetes.io/projected/a10485c9-afa9-4726-81d0-e7e45f1d91fd-kube-api-access-mhb82\") pod \"test-curl-7655-140249961882656\" (UID: \"a10485c9-afa9-4726-81d0-e7e45f1d91fd\") " pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:05.088194 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.088167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhb82\" (UniqueName: \"kubernetes.io/projected/a10485c9-afa9-4726-81d0-e7e45f1d91fd-kube-api-access-mhb82\") pod \"test-curl-7655-140249961882656\" (UID: \"a10485c9-afa9-4726-81d0-e7e45f1d91fd\") " pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:05.269884 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.269852 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:05.412241 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.412199 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcfae845-a07d-4c6f-90ab-f9c62ceabcd0" path="/var/lib/kubelet/pods/dcfae845-a07d-4c6f-90ab-f9c62ceabcd0/volumes" Jul 23 01:05:05.599893 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.599823 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961893424" Jul 23 01:05:05.599893 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.599840 2585 scope.go:117] "RemoveContainer" containerID="13abb1e2ba98e84772cc6d0f7a960fdb975f6d544f58947008f6cd0a8c0e6877" Jul 23 01:05:05.612858 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:05.612836 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:05.614096 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:05.614064 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda10485c9_afa9_4726_81d0_e7e45f1d91fd.slice/crio-d5a2810642883aa76552ce5d314f54885f4b3fe64550ea4503be285b9e9fe92c WatchSource:0}: Error finding container d5a2810642883aa76552ce5d314f54885f4b3fe64550ea4503be285b9e9fe92c: Status 404 returned error can't find the container with id d5a2810642883aa76552ce5d314f54885f4b3fe64550ea4503be285b9e9fe92c Jul 23 01:05:06.607504 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:06.607454 2585 generic.go:356] "Generic (PLEG): container finished" podID="a10485c9-afa9-4726-81d0-e7e45f1d91fd" containerID="43c95644275130c449ea1eb3d219c311103ed1d23f5d69459ecc33fb9b6b3309" exitCode=0 Jul 23 01:05:06.607974 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:06.607536 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961882656" event={"ID":"a10485c9-afa9-4726-81d0-e7e45f1d91fd","Type":"ContainerDied","Data":"43c95644275130c449ea1eb3d219c311103ed1d23f5d69459ecc33fb9b6b3309"} Jul 23 01:05:06.607974 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:06.607575 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961882656" event={"ID":"a10485c9-afa9-4726-81d0-e7e45f1d91fd","Type":"ContainerStarted","Data":"d5a2810642883aa76552ce5d314f54885f4b3fe64550ea4503be285b9e9fe92c"} Jul 23 01:05:07.751808 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.751783 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:07.786325 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.786297 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7655-140249961882656_a10485c9-afa9-4726-81d0-e7e45f1d91fd/test-curl-7655-140249961882656/0.log" Jul 23 01:05:07.805442 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.805412 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhb82\" (UniqueName: \"kubernetes.io/projected/a10485c9-afa9-4726-81d0-e7e45f1d91fd-kube-api-access-mhb82\") pod \"a10485c9-afa9-4726-81d0-e7e45f1d91fd\" (UID: \"a10485c9-afa9-4726-81d0-e7e45f1d91fd\") " Jul 23 01:05:07.807557 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.807531 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10485c9-afa9-4726-81d0-e7e45f1d91fd-kube-api-access-mhb82" (OuterVolumeSpecName: "kube-api-access-mhb82") pod "a10485c9-afa9-4726-81d0-e7e45f1d91fd" (UID: "a10485c9-afa9-4726-81d0-e7e45f1d91fd"). InnerVolumeSpecName "kube-api-access-mhb82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:07.816820 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.816788 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:07.820037 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.820015 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:07.906367 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.906280 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhb82\" (UniqueName: \"kubernetes.io/projected/a10485c9-afa9-4726-81d0-e7e45f1d91fd-kube-api-access-mhb82\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:07.950016 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.949979 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:07.950394 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.950379 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10485c9-afa9-4726-81d0-e7e45f1d91fd" containerName="test-curl-7655-140249961882656" Jul 23 01:05:07.950439 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.950395 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10485c9-afa9-4726-81d0-e7e45f1d91fd" containerName="test-curl-7655-140249961882656" Jul 23 01:05:07.950511 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.950501 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10485c9-afa9-4726-81d0-e7e45f1d91fd" containerName="test-curl-7655-140249961882656" Jul 23 01:05:07.953589 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.953573 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:07.961731 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:07.961705 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:08.007304 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.007273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd64g\" (UniqueName: \"kubernetes.io/projected/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7-kube-api-access-nd64g\") pod \"test-curl-7655-140249961882656\" (UID: \"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7\") " pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:08.108508 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.108441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd64g\" (UniqueName: \"kubernetes.io/projected/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7-kube-api-access-nd64g\") pod \"test-curl-7655-140249961882656\" (UID: \"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7\") " pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:08.118592 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.118564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd64g\" (UniqueName: \"kubernetes.io/projected/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7-kube-api-access-nd64g\") pod \"test-curl-7655-140249961882656\" (UID: \"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7\") " pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:08.264257 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.264225 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:08.401072 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.401041 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:08.403023 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:08.402995 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f5c2e6f_a5de_4c24_a96b_e50a37a908d7.slice/crio-02fcea55c0113ec7404ba6a99321f62815962e0634c4cf82c855de0c6ad2bcb0 WatchSource:0}: Error finding container 02fcea55c0113ec7404ba6a99321f62815962e0634c4cf82c855de0c6ad2bcb0: Status 404 returned error can't find the container with id 02fcea55c0113ec7404ba6a99321f62815962e0634c4cf82c855de0c6ad2bcb0 Jul 23 01:05:08.621144 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.621051 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961882656" event={"ID":"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7","Type":"ContainerStarted","Data":"02fcea55c0113ec7404ba6a99321f62815962e0634c4cf82c855de0c6ad2bcb0"} Jul 23 01:05:08.622228 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.622207 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:08.622345 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.622196 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5a2810642883aa76552ce5d314f54885f4b3fe64550ea4503be285b9e9fe92c" Jul 23 01:05:08.633630 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:08.633598 2585 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7655-140249961882656" oldPodUID="a10485c9-afa9-4726-81d0-e7e45f1d91fd" podUID="2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" Jul 23 01:05:09.412621 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:09.412582 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10485c9-afa9-4726-81d0-e7e45f1d91fd" path="/var/lib/kubelet/pods/a10485c9-afa9-4726-81d0-e7e45f1d91fd/volumes" Jul 23 01:05:09.630191 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:09.630158 2585 generic.go:356] "Generic (PLEG): container finished" podID="2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" containerID="8cd17f68bb4311d7b92f563d9239cd402d8c84e7144fe1483e7f14c1330756dc" exitCode=0 Jul 23 01:05:09.630333 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:09.630206 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961882656" event={"ID":"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7","Type":"ContainerDied","Data":"8cd17f68bb4311d7b92f563d9239cd402d8c84e7144fe1483e7f14c1330756dc"} Jul 23 01:05:10.770787 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.770754 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:10.789966 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.789943 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7655-140249961882656_2f5c2e6f-a5de-4c24-a96b-e50a37a908d7/test-curl-7655-140249961882656/0.log" Jul 23 01:05:10.815344 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.815307 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:10.818160 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.818135 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7655-140249961882656"] Jul 23 01:05:10.833231 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.833205 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd64g\" (UniqueName: \"kubernetes.io/projected/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7-kube-api-access-nd64g\") pod \"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7\" (UID: \"2f5c2e6f-a5de-4c24-a96b-e50a37a908d7\") " Jul 23 01:05:10.835361 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.835331 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7-kube-api-access-nd64g" (OuterVolumeSpecName: "kube-api-access-nd64g") pod "2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" (UID: "2f5c2e6f-a5de-4c24-a96b-e50a37a908d7"). InnerVolumeSpecName "kube-api-access-nd64g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:10.934857 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.934768 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd64g\" (UniqueName: \"kubernetes.io/projected/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7-kube-api-access-nd64g\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:10.949244 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.949214 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:10.949643 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.949631 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" containerName="test-curl-7655-140249961882656" Jul 23 01:05:10.949685 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.949646 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" containerName="test-curl-7655-140249961882656" Jul 23 01:05:10.949722 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.949714 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" containerName="test-curl-7655-140249961882656" Jul 23 01:05:10.952772 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.952758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:10.956772 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:10.956744 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:11.035387 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.035355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtjm\" (UniqueName: \"kubernetes.io/projected/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9-kube-api-access-nqtjm\") pod \"test-curl-7655-140249961883664\" (UID: \"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9\") " pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:11.136090 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.136057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtjm\" (UniqueName: \"kubernetes.io/projected/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9-kube-api-access-nqtjm\") pod \"test-curl-7655-140249961883664\" (UID: \"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9\") " pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:11.143355 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.143326 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtjm\" (UniqueName: \"kubernetes.io/projected/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9-kube-api-access-nqtjm\") pod \"test-curl-7655-140249961883664\" (UID: \"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9\") " pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:11.263581 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.263553 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:11.398341 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.398308 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:11.400281 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:11.400252 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e3ba10c_1814_44d4_a22c_b5e1ca1818f9.slice/crio-a51bf23cf927221f8736602d54f6124fe951e18e318115ca322a39a2d65785b7 WatchSource:0}: Error finding container a51bf23cf927221f8736602d54f6124fe951e18e318115ca322a39a2d65785b7: Status 404 returned error can't find the container with id a51bf23cf927221f8736602d54f6124fe951e18e318115ca322a39a2d65785b7 Jul 23 01:05:11.414412 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.414378 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5c2e6f-a5de-4c24-a96b-e50a37a908d7" path="/var/lib/kubelet/pods/2f5c2e6f-a5de-4c24-a96b-e50a37a908d7/volumes" Jul 23 01:05:11.642367 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.642293 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961882656" Jul 23 01:05:11.642367 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.642352 2585 scope.go:117] "RemoveContainer" containerID="8cd17f68bb4311d7b92f563d9239cd402d8c84e7144fe1483e7f14c1330756dc" Jul 23 01:05:11.643769 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:11.643655 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961883664" event={"ID":"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9","Type":"ContainerStarted","Data":"a51bf23cf927221f8736602d54f6124fe951e18e318115ca322a39a2d65785b7"} Jul 23 01:05:12.651097 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:12.651065 2585 generic.go:356] "Generic (PLEG): container finished" podID="7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" containerID="55a629925d10e42ffc1dca4287edf664986c789b937a92d7290f9e9c74f62074" exitCode=0 Jul 23 01:05:12.651504 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:12.651155 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961883664" event={"ID":"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9","Type":"ContainerDied","Data":"55a629925d10e42ffc1dca4287edf664986c789b937a92d7290f9e9c74f62074"} Jul 23 01:05:13.813136 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.813108 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:13.834060 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.834033 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7655-140249961883664_7e3ba10c-1814-44d4-a22c-b5e1ca1818f9/test-curl-7655-140249961883664/0.log" Jul 23 01:05:13.857294 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.857258 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:13.861114 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.861086 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:13.872586 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.872563 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqtjm\" (UniqueName: \"kubernetes.io/projected/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9-kube-api-access-nqtjm\") pod \"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9\" (UID: \"7e3ba10c-1814-44d4-a22c-b5e1ca1818f9\") " Jul 23 01:05:13.874599 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.874579 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9-kube-api-access-nqtjm" (OuterVolumeSpecName: "kube-api-access-nqtjm") pod "7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" (UID: "7e3ba10c-1814-44d4-a22c-b5e1ca1818f9"). InnerVolumeSpecName "kube-api-access-nqtjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:13.973727 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.973628 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqtjm\" (UniqueName: \"kubernetes.io/projected/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9-kube-api-access-nqtjm\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:13.987756 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.987727 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:13.988181 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.988162 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" containerName="test-curl-7655-140249961883664" Jul 23 01:05:13.988238 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.988184 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" containerName="test-curl-7655-140249961883664" Jul 23 01:05:13.988271 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.988256 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" containerName="test-curl-7655-140249961883664" Jul 23 01:05:13.991556 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.991538 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:13.998410 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:13.998381 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:14.074535 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.074498 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdwr\" (UniqueName: \"kubernetes.io/projected/c6dd1f62-c971-434a-996f-265878e680ea-kube-api-access-wmdwr\") pod \"test-curl-7655-140249961883664\" (UID: \"c6dd1f62-c971-434a-996f-265878e680ea\") " pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:14.175231 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.175196 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdwr\" (UniqueName: \"kubernetes.io/projected/c6dd1f62-c971-434a-996f-265878e680ea-kube-api-access-wmdwr\") pod \"test-curl-7655-140249961883664\" (UID: \"c6dd1f62-c971-434a-996f-265878e680ea\") " pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:14.184239 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.184205 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdwr\" (UniqueName: \"kubernetes.io/projected/c6dd1f62-c971-434a-996f-265878e680ea-kube-api-access-wmdwr\") pod \"test-curl-7655-140249961883664\" (UID: \"c6dd1f62-c971-434a-996f-265878e680ea\") " pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:14.302718 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.302674 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:14.441664 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.441637 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:14.443575 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:14.443545 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6dd1f62_c971_434a_996f_265878e680ea.slice/crio-14f9775a2f321c762c698dcc8cf0171aae7a501c8714f46f51b30579bee42a2f WatchSource:0}: Error finding container 14f9775a2f321c762c698dcc8cf0171aae7a501c8714f46f51b30579bee42a2f: Status 404 returned error can't find the container with id 14f9775a2f321c762c698dcc8cf0171aae7a501c8714f46f51b30579bee42a2f Jul 23 01:05:14.664150 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.664063 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:14.664150 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.664071 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a51bf23cf927221f8736602d54f6124fe951e18e318115ca322a39a2d65785b7" Jul 23 01:05:14.666158 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.666129 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961883664" event={"ID":"c6dd1f62-c971-434a-996f-265878e680ea","Type":"ContainerStarted","Data":"14f9775a2f321c762c698dcc8cf0171aae7a501c8714f46f51b30579bee42a2f"} Jul 23 01:05:14.676272 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:14.676250 2585 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7655-140249961883664" oldPodUID="7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" podUID="c6dd1f62-c971-434a-996f-265878e680ea" Jul 23 01:05:15.412632 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:15.412597 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3ba10c-1814-44d4-a22c-b5e1ca1818f9" path="/var/lib/kubelet/pods/7e3ba10c-1814-44d4-a22c-b5e1ca1818f9/volumes" Jul 23 01:05:15.673530 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:15.673396 2585 generic.go:356] "Generic (PLEG): container finished" podID="c6dd1f62-c971-434a-996f-265878e680ea" containerID="45bbb5d89017f8746bcaba69fd117947a032a2e9a114ab40272a1cf4a5f29992" exitCode=0 Jul 23 01:05:15.673530 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:15.673451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7655-140249961883664" event={"ID":"c6dd1f62-c971-434a-996f-265878e680ea","Type":"ContainerDied","Data":"45bbb5d89017f8746bcaba69fd117947a032a2e9a114ab40272a1cf4a5f29992"} Jul 23 01:05:16.819653 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:16.819629 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:16.839934 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:16.839902 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7655-140249961883664_c6dd1f62-c971-434a-996f-265878e680ea/test-curl-7655-140249961883664/0.log" Jul 23 01:05:16.863715 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:16.863686 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:16.865700 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:16.865680 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7655-140249961883664"] Jul 23 01:05:16.901479 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:16.901443 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmdwr\" (UniqueName: \"kubernetes.io/projected/c6dd1f62-c971-434a-996f-265878e680ea-kube-api-access-wmdwr\") pod \"c6dd1f62-c971-434a-996f-265878e680ea\" (UID: \"c6dd1f62-c971-434a-996f-265878e680ea\") " Jul 23 01:05:16.903648 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:16.903623 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dd1f62-c971-434a-996f-265878e680ea-kube-api-access-wmdwr" (OuterVolumeSpecName: "kube-api-access-wmdwr") pod "c6dd1f62-c971-434a-996f-265878e680ea" (UID: "c6dd1f62-c971-434a-996f-265878e680ea"). InnerVolumeSpecName "kube-api-access-wmdwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:17.003249 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.003207 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdwr\" (UniqueName: \"kubernetes.io/projected/c6dd1f62-c971-434a-996f-265878e680ea-kube-api-access-wmdwr\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:17.094414 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.094375 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz"] Jul 23 01:05:17.094825 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.094811 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6dd1f62-c971-434a-996f-265878e680ea" containerName="test-curl-7655-140249961883664" Jul 23 01:05:17.094874 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.094827 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6dd1f62-c971-434a-996f-265878e680ea" containerName="test-curl-7655-140249961883664" Jul 23 01:05:17.094914 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.094887 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6dd1f62-c971-434a-996f-265878e680ea" containerName="test-curl-7655-140249961883664" Jul 23 01:05:17.097939 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.097922 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:17.099995 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.099958 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:05:17.100110 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.100095 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 01:05:17.106076 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.105785 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz"] Jul 23 01:05:17.205424 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.205380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82f01e1-0cb7-4a2b-b412-9103ec16c79a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz\" (UID: \"a82f01e1-0cb7-4a2b-b412-9103ec16c79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:17.306319 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.306225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82f01e1-0cb7-4a2b-b412-9103ec16c79a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz\" (UID: \"a82f01e1-0cb7-4a2b-b412-9103ec16c79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:17.306924 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.306898 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82f01e1-0cb7-4a2b-b412-9103ec16c79a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz\" (UID: \"a82f01e1-0cb7-4a2b-b412-9103ec16c79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:17.411688 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.411654 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:17.411885 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.411686 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6dd1f62-c971-434a-996f-265878e680ea" path="/var/lib/kubelet/pods/c6dd1f62-c971-434a-996f-265878e680ea/volumes" Jul 23 01:05:17.550834 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.550806 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz"] Jul 23 01:05:17.552437 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:17.552412 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82f01e1_0cb7_4a2b_b412_9103ec16c79a.slice/crio-8118707561b78a03e999eb59a49d0e7dd1c8b9229561c742ec5448be2e266e61 WatchSource:0}: Error finding container 8118707561b78a03e999eb59a49d0e7dd1c8b9229561c742ec5448be2e266e61: Status 404 returned error can't find the container with id 8118707561b78a03e999eb59a49d0e7dd1c8b9229561c742ec5448be2e266e61 Jul 23 01:05:17.555587 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.555564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:17.657225 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.657196 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:17.687932 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.687905 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7655-140249961883664" Jul 23 01:05:17.688115 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.688012 2585 scope.go:117] "RemoveContainer" containerID="45bbb5d89017f8746bcaba69fd117947a032a2e9a114ab40272a1cf4a5f29992" Jul 23 01:05:17.690170 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:17.690064 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" event={"ID":"a82f01e1-0cb7-4a2b-b412-9103ec16c79a","Type":"ContainerStarted","Data":"8118707561b78a03e999eb59a49d0e7dd1c8b9229561c742ec5448be2e266e61"} Jul 23 01:05:18.699883 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:18.699849 2585 generic.go:356] "Generic (PLEG): container finished" podID="a82f01e1-0cb7-4a2b-b412-9103ec16c79a" containerID="849d3e56061bd164dbf3a913f54e4ffa1d95010db210f563a22774fd2bf1a8c5" exitCode=0 Jul 23 01:05:18.700292 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:18.699918 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" event={"ID":"a82f01e1-0cb7-4a2b-b412-9103ec16c79a","Type":"ContainerDied","Data":"849d3e56061bd164dbf3a913f54e4ffa1d95010db210f563a22774fd2bf1a8c5"} Jul 23 01:05:19.846548 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:19.846523 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:19.931544 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:19.931508 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82f01e1-0cb7-4a2b-b412-9103ec16c79a-maas-api-service-ca\") pod \"a82f01e1-0cb7-4a2b-b412-9103ec16c79a\" (UID: \"a82f01e1-0cb7-4a2b-b412-9103ec16c79a\") " Jul 23 01:05:19.931912 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:19.931887 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82f01e1-0cb7-4a2b-b412-9103ec16c79a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a82f01e1-0cb7-4a2b-b412-9103ec16c79a" (UID: "a82f01e1-0cb7-4a2b-b412-9103ec16c79a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:05:20.032413 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:20.032380 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82f01e1-0cb7-4a2b-b412-9103ec16c79a-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:20.712238 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:20.712206 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" Jul 23 01:05:20.712389 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:20.712243 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz" event={"ID":"a82f01e1-0cb7-4a2b-b412-9103ec16c79a","Type":"ContainerDied","Data":"8118707561b78a03e999eb59a49d0e7dd1c8b9229561c742ec5448be2e266e61"} Jul 23 01:05:20.712389 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:20.712294 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8118707561b78a03e999eb59a49d0e7dd1c8b9229561c742ec5448be2e266e61" Jul 23 01:05:27.513846 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:27.513813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:27.514742 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:27.514721 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q"] Jul 23 01:05:27.518221 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:27.518196 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b90c57-29746140-nvc7q"] Jul 23 01:05:29.411829 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:29.411799 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e26d2d-afce-42c8-8d76-87b58ac398a1" path="/var/lib/kubelet/pods/42e26d2d-afce-42c8-8d76-87b58ac398a1/volumes" Jul 23 01:05:32.145223 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:32.145187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:32.163086 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:32.163054 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz"] Jul 23 01:05:32.165297 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:32.165278 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b90c57-nb8xz"] Jul 23 01:05:32.576298 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:32.576269 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:33.413527 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:33.413485 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82f01e1-0cb7-4a2b-b412-9103ec16c79a" path="/var/lib/kubelet/pods/a82f01e1-0cb7-4a2b-b412-9103ec16c79a/volumes" Jul 23 01:05:38.788726 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:38.788687 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:38.889751 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:38.889711 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:39.052165 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.052082 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm"] Jul 23 01:05:39.052318 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.052296 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" containerID="cri-o://ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99" gracePeriod=30 Jul 23 01:05:39.075194 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.075150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:39.331183 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.331088 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:39.451673 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.451642 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg"] Jul 23 01:05:39.452042 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.452030 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82f01e1-0cb7-4a2b-b412-9103ec16c79a" containerName="revoke-keys" Jul 23 01:05:39.452090 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.452044 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82f01e1-0cb7-4a2b-b412-9103ec16c79a" containerName="revoke-keys" Jul 23 01:05:39.452137 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.452128 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82f01e1-0cb7-4a2b-b412-9103ec16c79a" containerName="revoke-keys" Jul 23 01:05:39.457034 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.457015 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:39.459104 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.459085 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:05:39.459264 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.459245 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d488k\"" Jul 23 01:05:39.462767 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.462738 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg"] Jul 23 01:05:39.518772 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.518732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6551316-77eb-48ec-a168-72bd0295b0f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg\" (UID: \"c6551316-77eb-48ec-a168-72bd0295b0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:39.619311 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.619219 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6551316-77eb-48ec-a168-72bd0295b0f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg\" (UID: \"c6551316-77eb-48ec-a168-72bd0295b0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:39.619957 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.619936 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6551316-77eb-48ec-a168-72bd0295b0f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg\" (UID: \"c6551316-77eb-48ec-a168-72bd0295b0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:39.768763 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.768722 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:39.913421 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.913392 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg"] Jul 23 01:05:39.916019 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:05:39.915992 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6551316_77eb_48ec_a168_72bd0295b0f3.slice/crio-10f33e63a1d6fb8e60f6094b2c87cb962998b7c46d7a35df7a363ec4dfba315e WatchSource:0}: Error finding container 10f33e63a1d6fb8e60f6094b2c87cb962998b7c46d7a35df7a363ec4dfba315e: Status 404 returned error can't find the container with id 10f33e63a1d6fb8e60f6094b2c87cb962998b7c46d7a35df7a363ec4dfba315e Jul 23 01:05:39.949613 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:39.949589 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:40.049770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:40.049740 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:40.834290 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:40.834253 2585 generic.go:356] "Generic (PLEG): container finished" podID="c6551316-77eb-48ec-a168-72bd0295b0f3" containerID="093a8e74d007edb21d383be24c1408b2a642a3b47731266e4d40613eb8408154" exitCode=0 Jul 23 01:05:40.834514 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:40.834337 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" event={"ID":"c6551316-77eb-48ec-a168-72bd0295b0f3","Type":"ContainerDied","Data":"093a8e74d007edb21d383be24c1408b2a642a3b47731266e4d40613eb8408154"} Jul 23 01:05:40.834514 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:40.834441 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" event={"ID":"c6551316-77eb-48ec-a168-72bd0295b0f3","Type":"ContainerStarted","Data":"10f33e63a1d6fb8e60f6094b2c87cb962998b7c46d7a35df7a363ec4dfba315e"} Jul 23 01:05:41.993340 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:41.993315 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:42.042102 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:42.042065 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6551316-77eb-48ec-a168-72bd0295b0f3-maas-api-service-ca\") pod \"c6551316-77eb-48ec-a168-72bd0295b0f3\" (UID: \"c6551316-77eb-48ec-a168-72bd0295b0f3\") " Jul 23 01:05:42.042370 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:42.042341 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6551316-77eb-48ec-a168-72bd0295b0f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6551316-77eb-48ec-a168-72bd0295b0f3" (UID: "c6551316-77eb-48ec-a168-72bd0295b0f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:05:42.143272 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:42.143188 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6551316-77eb-48ec-a168-72bd0295b0f3-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:42.852346 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:42.852315 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" Jul 23 01:05:42.852543 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:42.852351 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg" event={"ID":"c6551316-77eb-48ec-a168-72bd0295b0f3","Type":"ContainerDied","Data":"10f33e63a1d6fb8e60f6094b2c87cb962998b7c46d7a35df7a363ec4dfba315e"} Jul 23 01:05:42.852543 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:42.852399 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f33e63a1d6fb8e60f6094b2c87cb962998b7c46d7a35df7a363ec4dfba315e" Jul 23 01:05:44.316742 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.316672 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 01:05:44.363566 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363541 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a5e059-9e4b-4476-884f-f6b3437728ae-istiod-ca-cert\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.363762 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363583 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-data\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.363762 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363611 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49f5z\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-kube-api-access-49f5z\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.363762 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363644 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-envoy\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.363762 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363675 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-credential-socket\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.363762 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363699 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-socket\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.363762 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363742 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-podinfo\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.364084 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363812 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-certs\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.364084 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363876 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-token\") pod \"65a5e059-9e4b-4476-884f-f6b3437728ae\" (UID: \"65a5e059-9e4b-4476-884f-f6b3437728ae\") " Jul 23 01:05:44.364084 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.363955 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a5e059-9e4b-4476-884f-f6b3437728ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:05:44.364084 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.364012 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:44.364301 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.364218 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a5e059-9e4b-4476-884f-f6b3437728ae-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.364301 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.364237 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.364301 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.364279 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:44.364444 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.364315 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:44.364651 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.364621 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:44.366070 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.366034 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-kube-api-access-49f5z" (OuterVolumeSpecName: "kube-api-access-49f5z") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "kube-api-access-49f5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:44.366182 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.366113 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:44.366588 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.366563 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:44.366674 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.366586 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65a5e059-9e4b-4476-884f-f6b3437728ae" (UID: "65a5e059-9e4b-4476-884f-f6b3437728ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 01:05:44.465065 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465022 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.465065 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465058 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.465065 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465067 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.465065 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465076 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.465387 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465086 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.465387 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465094 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f5z\" (UniqueName: \"kubernetes.io/projected/65a5e059-9e4b-4476-884f-f6b3437728ae-kube-api-access-49f5z\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.465387 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.465102 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a5e059-9e4b-4476-884f-f6b3437728ae-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:05:44.864357 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.864320 2585 generic.go:356] "Generic (PLEG): container finished" podID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerID="ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99" exitCode=0 Jul 23 01:05:44.864546 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.864375 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" Jul 23 01:05:44.864546 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.864412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" event={"ID":"65a5e059-9e4b-4476-884f-f6b3437728ae","Type":"ContainerDied","Data":"ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99"} Jul 23 01:05:44.864546 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.864491 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm" event={"ID":"65a5e059-9e4b-4476-884f-f6b3437728ae","Type":"ContainerDied","Data":"ff2f33cf5748100f4ddca1c5d7ece508388d69178cd034a0ff6f0d3246d8d617"} Jul 23 01:05:44.864546 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.864524 2585 scope.go:117] "RemoveContainer" containerID="ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99" Jul 23 01:05:44.874813 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.874793 2585 scope.go:117] "RemoveContainer" containerID="3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de" Jul 23 01:05:44.883304 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.883283 2585 scope.go:117] "RemoveContainer" containerID="ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99" Jul 23 01:05:44.883711 ip-10-0-139-142 kubenswrapper[2585]: E0723 01:05:44.883691 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99\": container with ID starting with ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99 not found: ID does not exist" containerID="ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99" Jul 23 01:05:44.883814 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.883721 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99"} err="failed to get container status \"ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99\": rpc error: code = NotFound desc = could not find container \"ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99\": container with ID starting with ac5703cb0afb8d4d8c0bb5296a49c44315a25347716fcc5a81bc13a9d8315b99 not found: ID does not exist" Jul 23 01:05:44.883814 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.883744 2585 scope.go:117] "RemoveContainer" containerID="3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de" Jul 23 01:05:44.884047 ip-10-0-139-142 kubenswrapper[2585]: E0723 01:05:44.884025 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de\": container with ID starting with 3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de not found: ID does not exist" containerID="3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de" Jul 23 01:05:44.884091 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.884053 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de"} err="failed to get container status \"3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de\": rpc error: code = NotFound desc = could not find container \"3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de\": container with ID starting with 3f39cab2ad3f229e6f2a730576756b25492e4a3d046e5486727d08d3d56030de not found: ID does not exist" Jul 23 01:05:44.885615 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.885596 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm"] Jul 23 01:05:44.889024 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:44.889006 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b90c57-openshift-default-dd4c5d7ff-bcqjm"] Jul 23 01:05:45.411401 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:45.411367 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" path="/var/lib/kubelet/pods/65a5e059-9e4b-4476-884f-f6b3437728ae/volumes" Jul 23 01:05:49.971054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:49.971006 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b"] Jul 23 01:05:49.973230 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:49.973200 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-75b665-29746140-lqw9b"] Jul 23 01:05:49.974023 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:49.973991 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:51.411354 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:51.411324 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7094f88c-9d60-4fec-9c00-2d6f09b93035" path="/var/lib/kubelet/pods/7094f88c-9d60-4fec-9c00-2d6f09b93035/volumes" Jul 23 01:05:54.521621 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:54.521527 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg"] Jul 23 01:05:54.522194 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:54.522161 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:54.524108 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:54.524088 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-75b665-bc8zg"] Jul 23 01:05:54.910015 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:54.909931 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:55.413076 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:05:55.413040 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6551316-77eb-48ec-a168-72bd0295b0f3" path="/var/lib/kubelet/pods/c6551316-77eb-48ec-a168-72bd0295b0f3/volumes" Jul 23 01:06:00.813507 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:00.813449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:00.882834 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:00.882792 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:01.071014 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:01.070927 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s"] Jul 23 01:06:01.071235 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:01.071194 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" containerID="cri-o://d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f" gracePeriod=30 Jul 23 01:06:01.094404 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:01.094375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:01.340834 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:01.340698 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:02.323987 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:02.323956 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:06.330000 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.329968 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 01:06:06.478690 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478603 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-envoy\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478690 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478647 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-credential-socket\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478690 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478683 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istiod-ca-cert\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478978 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478711 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-socket\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478978 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478731 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-certs\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478978 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478759 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6ns8\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-kube-api-access-d6ns8\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478978 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478800 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-podinfo\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478978 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478832 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-data\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.478978 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.478874 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-token\") pod \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\" (UID: \"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e\") " Jul 23 01:06:06.479297 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.479149 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:06:06.479297 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.479165 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:06:06.479297 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.479160 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:06:06.479458 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.479310 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:06:06.480269 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.480239 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:06:06.481124 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.481097 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:06:06.481265 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.481247 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:06:06.481336 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.481314 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-kube-api-access-d6ns8" (OuterVolumeSpecName: "kube-api-access-d6ns8") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "kube-api-access-d6ns8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:06:06.481336 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.481318 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" (UID: "6ad146f8-43b8-4c0b-87b7-7db166f5ac2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.579951 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.579989 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ns8\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-kube-api-access-d6ns8\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580003 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580015 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580028 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580040 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580051 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580064 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.580599 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.580077 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Jul 23 01:06:06.989685 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.989647 2585 generic.go:356] "Generic (PLEG): container finished" podID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerID="d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f" exitCode=0 Jul 23 01:06:06.989856 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.989703 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" Jul 23 01:06:06.989856 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.989739 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" event={"ID":"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e","Type":"ContainerDied","Data":"d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f"} Jul 23 01:06:06.989856 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.989795 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s" event={"ID":"6ad146f8-43b8-4c0b-87b7-7db166f5ac2e","Type":"ContainerDied","Data":"c0c2d8175e6ab9a8b87ae0369cbefa75d155a77d2cfd4e0e97a479746b83b140"} Jul 23 01:06:06.989856 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.989821 2585 scope.go:117] "RemoveContainer" containerID="d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f" Jul 23 01:06:06.999290 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:06.999272 2585 scope.go:117] "RemoveContainer" containerID="d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9" Jul 23 01:06:07.007924 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.007907 2585 scope.go:117] "RemoveContainer" containerID="d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f" Jul 23 01:06:07.008157 ip-10-0-139-142 kubenswrapper[2585]: E0723 01:06:07.008141 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f\": container with ID starting with d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f not found: ID does not exist" containerID="d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f" Jul 23 01:06:07.008224 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.008164 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f"} err="failed to get container status \"d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f\": rpc error: code = NotFound desc = could not find container \"d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f\": container with ID starting with d03db3205335920470f1293711633f6e2d18dafa1177f1bfc06b292ab25b882f not found: ID does not exist" Jul 23 01:06:07.008224 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.008179 2585 scope.go:117] "RemoveContainer" containerID="d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9" Jul 23 01:06:07.008384 ip-10-0-139-142 kubenswrapper[2585]: E0723 01:06:07.008369 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9\": container with ID starting with d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9 not found: ID does not exist" containerID="d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9" Jul 23 01:06:07.008425 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.008386 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9"} err="failed to get container status \"d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9\": rpc error: code = NotFound desc = could not find container \"d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9\": container with ID starting with d449c0c86628fe81cbdd77be2e3b6b59c59184bb855b885bf1d7e22d28f566e9 not found: ID does not exist" Jul 23 01:06:07.011313 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.011292 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s"] Jul 23 01:06:07.015967 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.015947 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-75b665-openshift-default-54c868597f-62r6s"] Jul 23 01:06:07.411350 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.411274 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" path="/var/lib/kubelet/pods/6ad146f8-43b8-4c0b-87b7-7db166f5ac2e/volumes" Jul 23 01:06:07.492523 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.492498 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kzldj_d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9/manager/0.log" Jul 23 01:06:07.761653 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.761626 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jqk76_5bf6f6ed-da14-4dbe-a62f-c2207c069042/server/0.log" Jul 23 01:06:07.900552 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:07.900521 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g8mtv_d7067c92-8c8e-4880-93f8-2d88cecd4009/manager/0.log" Jul 23 01:06:08.288284 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:08.288259 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8844c5db8-fzf9m_0ed1f769-bc32-473e-b7b0-245754cc103a/copy-manifests/0.log" Jul 23 01:06:08.295270 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:08.295248 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8844c5db8-fzf9m_0ed1f769-bc32-473e-b7b0-245754cc103a/manager/0.log" Jul 23 01:06:08.762822 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:08.762796 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746110-nf9jm_16c31eb0-6b2a-4dc8-8dcc-341fecff97d5/cleanup/0.log" Jul 23 01:06:08.879437 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:08.879411 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746125-4c4gp_5113eabc-e5b6-4685-8d3f-e860e31befdd/cleanup/0.log" Jul 23 01:06:08.991249 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:08.991222 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746140-xp7rh_120ef1c7-ca0e-43ab-adad-2be678e90359/cleanup/0.log" Jul 23 01:06:10.605219 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:10.605189 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2rznc_d087e1fe-464c-4284-9ebd-7bdf767ad56b/manager/0.log" Jul 23 01:06:10.722388 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:10.722354 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pcndc_acd91885-e0e5-4c18-9506-1e6d6d967dba/manager/0.log" Jul 23 01:06:10.966649 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:10.966524 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2dh4f_cfe51fe7-3af6-4467-b549-5c3a8f478d01/registry-server/0.log" Jul 23 01:06:11.108118 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:11.108091 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vsbdj_381f3239-c898-46f2-b4e9-84716f61d46d/manager/0.log" Jul 23 01:06:11.398948 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:11.398919 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nzc77_c26161d7-a9d3-4bb6-b82b-90d5b43bae72/manager/0.log" Jul 23 01:06:11.766952 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:11.766909 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hcwkh_4853630f-e743-4b5d-a3a1-919e3280e496/istio-proxy/0.log" Jul 23 01:06:12.031153 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:12.031073 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c7d578648-5x8gg_9d05b2c8-049b-427a-ad50-68cdc2e6068b/kube-auth-proxy/0.log" Jul 23 01:06:12.275274 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:12.275228 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/1.log" Jul 23 01:06:12.650039 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:12.650001 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6fc97d76-kshb5_5fca11ac-2e0f-4249-9bfd-060eeae9dbd3/router/0.log" Jul 23 01:06:13.142187 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:13.142148 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-cwzvj_9d696b5f-f8f1-44f3-8fae-3fd32cecc6c6/main/0.log" Jul 23 01:06:19.534235 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534203 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rz22l/must-gather-bsnlq"] Jul 23 01:06:19.534639 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534593 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" Jul 23 01:06:19.534639 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534605 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" Jul 23 01:06:19.534639 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534617 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6551316-77eb-48ec-a168-72bd0295b0f3" containerName="revoke-keys" Jul 23 01:06:19.534639 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534624 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6551316-77eb-48ec-a168-72bd0295b0f3" containerName="revoke-keys" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534645 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534650 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534660 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534665 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534720 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534729 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534734 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6551316-77eb-48ec-a168-72bd0295b0f3" containerName="revoke-keys" Jul 23 01:06:19.534770 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534743 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad146f8-43b8-4c0b-87b7-7db166f5ac2e" containerName="istio-proxy" Jul 23 01:06:19.535008 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534799 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" Jul 23 01:06:19.535008 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534804 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" Jul 23 01:06:19.535008 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.534865 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a5e059-9e4b-4476-884f-f6b3437728ae" containerName="istio-proxy" Jul 23 01:06:19.538030 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.538009 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.540217 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.540200 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rz22l\"/\"openshift-service-ca.crt\"" Jul 23 01:06:19.540738 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.540721 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rz22l\"/\"default-dockercfg-htbdw\"" Jul 23 01:06:19.540824 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.540758 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rz22l\"/\"kube-root-ca.crt\"" Jul 23 01:06:19.548443 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.548422 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:19.548586 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.548551 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rz22l/must-gather-bsnlq"] Jul 23 01:06:19.611017 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.610979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:19.696305 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.696265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdh56\" (UniqueName: \"kubernetes.io/projected/43b1dade-bd07-4018-9412-518f4e877892-kube-api-access-tdh56\") pod \"must-gather-bsnlq\" (UID: \"43b1dade-bd07-4018-9412-518f4e877892\") " pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.696537 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.696314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43b1dade-bd07-4018-9412-518f4e877892-must-gather-output\") pod \"must-gather-bsnlq\" (UID: \"43b1dade-bd07-4018-9412-518f4e877892\") " pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.797227 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.797125 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdh56\" (UniqueName: \"kubernetes.io/projected/43b1dade-bd07-4018-9412-518f4e877892-kube-api-access-tdh56\") pod \"must-gather-bsnlq\" (UID: \"43b1dade-bd07-4018-9412-518f4e877892\") " pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.797227 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.797180 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43b1dade-bd07-4018-9412-518f4e877892-must-gather-output\") pod \"must-gather-bsnlq\" (UID: \"43b1dade-bd07-4018-9412-518f4e877892\") " pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.797576 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.797559 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43b1dade-bd07-4018-9412-518f4e877892-must-gather-output\") pod \"must-gather-bsnlq\" (UID: \"43b1dade-bd07-4018-9412-518f4e877892\") " pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.804478 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.804447 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdh56\" (UniqueName: \"kubernetes.io/projected/43b1dade-bd07-4018-9412-518f4e877892-kube-api-access-tdh56\") pod \"must-gather-bsnlq\" (UID: \"43b1dade-bd07-4018-9412-518f4e877892\") " pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.848231 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.848184 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rz22l/must-gather-bsnlq" Jul 23 01:06:19.983554 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:19.983524 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rz22l/must-gather-bsnlq"] Jul 23 01:06:19.986299 ip-10-0-139-142 kubenswrapper[2585]: W0723 01:06:19.986272 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43b1dade_bd07_4018_9412_518f4e877892.slice/crio-6547a3521c97ffce12bb469c804e473a52d605f8ea10305938544f0bc6da03f5 WatchSource:0}: Error finding container 6547a3521c97ffce12bb469c804e473a52d605f8ea10305938544f0bc6da03f5: Status 404 returned error can't find the container with id 6547a3521c97ffce12bb469c804e473a52d605f8ea10305938544f0bc6da03f5 Jul 23 01:06:20.060198 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:20.060107 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rz22l/must-gather-bsnlq" event={"ID":"43b1dade-bd07-4018-9412-518f4e877892","Type":"ContainerStarted","Data":"6547a3521c97ffce12bb469c804e473a52d605f8ea10305938544f0bc6da03f5"} Jul 23 01:06:21.069822 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:21.069794 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rz22l/must-gather-bsnlq" event={"ID":"43b1dade-bd07-4018-9412-518f4e877892","Type":"ContainerStarted","Data":"41edd8d344fc56b4d03a18b9ee7354fb8f927ffd725670b861dc5842039e4fb6"} Jul 23 01:06:22.079057 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:22.079024 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rz22l/must-gather-bsnlq" event={"ID":"43b1dade-bd07-4018-9412-518f4e877892","Type":"ContainerStarted","Data":"48ff3c8634f03020fbe53286d6de43f009ba4812ef2f30832ddd986200076c97"} Jul 23 01:06:22.096065 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:22.096010 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rz22l/must-gather-bsnlq" podStartSLOduration=2.201915284 podStartE2EDuration="3.09599017s" totalImagesPullingTime="894.074886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:06:22.094164161 +0000 UTC m=+2811.390859149" watchObservedRunningTime="2026-07-23 01:06:22.09599017 +0000 UTC m=+2811.392685139" Jul 23 01:06:22.536400 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:22.536371 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-spxkd_cd3be28b-aa8d-4a5a-b230-dbedffd5543c/global-pull-secret-syncer/0.log" Jul 23 01:06:22.591415 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:22.591370 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4z5pz_dd35f018-be61-4835-9c93-056a04942616/konnectivity-agent/0.log" Jul 23 01:06:22.700302 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:22.700269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-142.ec2.internal_42a5bd99e2334217167eb496f79450e6/haproxy/0.log" Jul 23 01:06:27.830641 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:27.829510 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2rznc_d087e1fe-464c-4284-9ebd-7bdf767ad56b/manager/0.log" Jul 23 01:06:27.865126 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:27.864951 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pcndc_acd91885-e0e5-4c18-9506-1e6d6d967dba/manager/0.log" Jul 23 01:06:27.960486 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:27.959936 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2dh4f_cfe51fe7-3af6-4467-b549-5c3a8f478d01/registry-server/0.log" Jul 23 01:06:28.156580 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:28.156497 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vsbdj_381f3239-c898-46f2-b4e9-84716f61d46d/manager/0.log" Jul 23 01:06:28.268797 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:28.268580 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nzc77_c26161d7-a9d3-4bb6-b82b-90d5b43bae72/manager/0.log" Jul 23 01:06:29.967027 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:29.966985 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-ndsgc_48d9735a-9a29-47c3-a62c-6626b8f341b2/cluster-monitoring-operator/0.log" Jul 23 01:06:30.242920 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:30.242886 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9xvd_5a464fcd-5763-44fe-8f3b-c7a7cf8b5709/node-exporter/0.log" Jul 23 01:06:30.272890 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:30.272859 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9xvd_5a464fcd-5763-44fe-8f3b-c7a7cf8b5709/kube-rbac-proxy/0.log" Jul 23 01:06:30.295783 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:30.295679 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9xvd_5a464fcd-5763-44fe-8f3b-c7a7cf8b5709/init-textfile/0.log" Jul 23 01:06:30.712523 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:30.712425 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-mhwz6_ed447874-471c-41d7-ae97-8b505d19dcd9/prometheus-operator-admission-webhook/0.log" Jul 23 01:06:31.704387 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:31.704351 2585 scope.go:117] "RemoveContainer" containerID="ba4502ff475c4c79e43634ee9ec00989197335d45d29ed314adf6d92b1a0b8e4" Jul 23 01:06:31.716786 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:31.716764 2585 scope.go:117] "RemoveContainer" containerID="76d74d49e06246425c46ce456bb6e4004764831cf4c7c38a7349f064442c7b57" Jul 23 01:06:32.162271 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.162246 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-gxpl4_417d7eb2-478f-44cd-aed6-554556aa68d8/networking-console-plugin/0.log" Jul 23 01:06:32.459308 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.459212 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd"] Jul 23 01:06:32.480701 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.480668 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd"] Jul 23 01:06:32.481045 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.481026 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.546491 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.544782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-sys\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.546491 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.544837 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-proc\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.546491 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.544881 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gh8\" (UniqueName: \"kubernetes.io/projected/abd33d2c-8b13-4e63-92fc-7878d52aba39-kube-api-access-b8gh8\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.546491 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.544913 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-lib-modules\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.546491 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.545035 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-podres\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.646522 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646477 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-podres\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.646720 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646544 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-sys\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.646720 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-proc\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.646720 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gh8\" (UniqueName: \"kubernetes.io/projected/abd33d2c-8b13-4e63-92fc-7878d52aba39-kube-api-access-b8gh8\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.646720 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646632 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-podres\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.646720 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-lib-modules\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.647050 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646717 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-proc\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.647050 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646905 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-lib-modules\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.647050 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.646660 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abd33d2c-8b13-4e63-92fc-7878d52aba39-sys\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.657672 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.657639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gh8\" (UniqueName: \"kubernetes.io/projected/abd33d2c-8b13-4e63-92fc-7878d52aba39-kube-api-access-b8gh8\") pod \"perf-node-gather-daemonset-qqsdd\" (UID: \"abd33d2c-8b13-4e63-92fc-7878d52aba39\") " pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.798526 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.798452 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:32.830543 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.830437 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/1.log" Jul 23 01:06:32.837284 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:32.837210 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wvjzl_92dd474f-1612-4464-ae50-4a24c828c187/console-operator/2.log" Jul 23 01:06:33.044943 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:33.044441 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd"] Jul 23 01:06:33.179055 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:33.179018 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" event={"ID":"abd33d2c-8b13-4e63-92fc-7878d52aba39","Type":"ContainerStarted","Data":"75cfacf14f3a738f9b31f3921adcdcfc7fbd3bc33fcf4d8d7a417be370c9c6c6"} Jul 23 01:06:33.494576 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:33.494547 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-2x84q_1a029abf-df82-48a2-aea2-531b0b24d44b/download-server/0.log" Jul 23 01:06:34.108892 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:34.108863 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-jh4tw_b28cf90a-3973-46bb-adff-d83bfdaad30c/volume-data-source-validator/0.log" Jul 23 01:06:34.188999 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:34.188957 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" event={"ID":"abd33d2c-8b13-4e63-92fc-7878d52aba39","Type":"ContainerStarted","Data":"ca7f5d7fa2476d26cb9135ae59ab7165e2f7472263aac9b1d565174f64f21de7"} Jul 23 01:06:34.189525 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:34.189444 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:34.205698 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:34.205649 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" podStartSLOduration=2.205632451 podStartE2EDuration="2.205632451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:06:34.204340947 +0000 UTC m=+2823.501035916" watchObservedRunningTime="2026-07-23 01:06:34.205632451 +0000 UTC m=+2823.502327430" Jul 23 01:06:35.206442 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:35.206416 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhz52_a40c425e-9a51-48a3-ad58-96eda4dddb66/dns/0.log" Jul 23 01:06:35.228217 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:35.228184 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhz52_a40c425e-9a51-48a3-ad58-96eda4dddb66/kube-rbac-proxy/0.log" Jul 23 01:06:35.335022 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:35.334994 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l4m8c_e1146a88-38cb-480f-af98-ead7a0b8a6b1/dns-node-resolver/0.log" Jul 23 01:06:35.924041 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:35.924006 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j5x4l_aa64cb4e-3578-4705-addb-932930f73df9/node-ca/0.log" Jul 23 01:06:36.933834 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:36.933808 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hcwkh_4853630f-e743-4b5d-a3a1-919e3280e496/istio-proxy/0.log" Jul 23 01:06:37.088564 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:37.088529 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c7d578648-5x8gg_9d05b2c8-049b-427a-ad50-68cdc2e6068b/kube-auth-proxy/0.log" Jul 23 01:06:37.150929 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:37.150894 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/0.log" Jul 23 01:06:37.187524 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:37.187411 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/1.log" Jul 23 01:06:37.273428 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:37.273397 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6fc97d76-kshb5_5fca11ac-2e0f-4249-9bfd-060eeae9dbd3/router/0.log" Jul 23 01:06:37.983849 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:37.983816 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qj2kb_7e966968-8137-444e-aaf0-e811d4fff488/serve-healthcheck-canary/0.log" Jul 23 01:06:38.451306 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:38.451228 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-8kf92_578cdcb0-4740-4a05-9466-3273fd0e990b/insights-operator/0.log" Jul 23 01:06:38.453012 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:38.452988 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-8kf92_578cdcb0-4740-4a05-9466-3273fd0e990b/insights-operator/1.log" Jul 23 01:06:38.614660 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:38.614635 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vjbl8_3a1deeea-5ee1-46da-8634-c754a6a24cc0/kube-rbac-proxy/0.log" Jul 23 01:06:38.636054 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:38.636028 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vjbl8_3a1deeea-5ee1-46da-8634-c754a6a24cc0/exporter/0.log" Jul 23 01:06:38.659229 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:38.659202 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vjbl8_3a1deeea-5ee1-46da-8634-c754a6a24cc0/extractor/0.log" Jul 23 01:06:40.209690 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:40.209651 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rz22l/perf-node-gather-daemonset-qqsdd" Jul 23 01:06:41.014949 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:41.014879 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kzldj_d7a1c9e3-2de3-40e1-b34f-dc8a8e10abb9/manager/0.log" Jul 23 01:06:41.148365 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:41.148328 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jqk76_5bf6f6ed-da14-4dbe-a62f-c2207c069042/server/0.log" Jul 23 01:06:41.252797 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:41.252767 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g8mtv_d7067c92-8c8e-4880-93f8-2d88cecd4009/manager/0.log" Jul 23 01:06:41.444526 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:41.444407 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8844c5db8-fzf9m_0ed1f769-bc32-473e-b7b0-245754cc103a/manager/0.log" Jul 23 01:06:41.467363 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:41.467330 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8844c5db8-fzf9m_0ed1f769-bc32-473e-b7b0-245754cc103a/copy-manifests/0.log" Jul 23 01:06:43.155813 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:43.155779 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hcwkh_4853630f-e743-4b5d-a3a1-919e3280e496/istio-proxy/0.log" Jul 23 01:06:43.281837 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:43.281809 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c7d578648-5x8gg_9d05b2c8-049b-427a-ad50-68cdc2e6068b/kube-auth-proxy/0.log" Jul 23 01:06:43.322698 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:43.322669 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/0.log" Jul 23 01:06:43.362898 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:43.362862 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/1.log" Jul 23 01:06:43.424795 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:43.424709 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6fc97d76-kshb5_5fca11ac-2e0f-4249-9bfd-060eeae9dbd3/router/0.log" Jul 23 01:06:44.605065 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:44.605033 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hcwkh_4853630f-e743-4b5d-a3a1-919e3280e496/istio-proxy/0.log" Jul 23 01:06:44.745655 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:44.745621 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c7d578648-5x8gg_9d05b2c8-049b-427a-ad50-68cdc2e6068b/kube-auth-proxy/0.log" Jul 23 01:06:44.784375 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:44.784331 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/0.log" Jul 23 01:06:44.820675 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:44.820584 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-w7s5z_04fc3c40-0734-4a37-ad19-85cbbed76224/istio-proxy/1.log" Jul 23 01:06:44.894677 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:44.894649 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6fc97d76-kshb5_5fca11ac-2e0f-4249-9bfd-060eeae9dbd3/router/0.log" Jul 23 01:06:47.923940 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:47.923912 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-kc5d4_93a735b7-a90b-4e28-bea9-1c484551f18e/migrator/0.log" Jul 23 01:06:47.945396 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:47.945372 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-kc5d4_93a735b7-a90b-4e28-bea9-1c484551f18e/graceful-termination/0.log" Jul 23 01:06:48.324350 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:48.324307 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mc4fh_b4ae9351-6cb1-4305-8e63-bca4ee681df1/kube-storage-version-migrator-operator/1.log" Jul 23 01:06:48.325378 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:48.325357 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mc4fh_b4ae9351-6cb1-4305-8e63-bca4ee681df1/kube-storage-version-migrator-operator/0.log" Jul 23 01:06:49.698308 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.698282 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/kube-multus-additional-cni-plugins/0.log" Jul 23 01:06:49.721115 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.721088 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/egress-router-binary-copy/0.log" Jul 23 01:06:49.742206 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.742184 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/cni-plugins/0.log" Jul 23 01:06:49.767105 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.767078 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/bond-cni-plugin/0.log" Jul 23 01:06:49.789356 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.789329 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/routeoverride-cni/0.log" Jul 23 01:06:49.810686 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.810657 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/whereabouts-cni-bincopy/0.log" Jul 23 01:06:49.833220 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.833192 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vsxtl_85950f20-fa7b-4aa1-9767-2dde5fd203ef/whereabouts-cni/0.log" Jul 23 01:06:49.903191 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.903118 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r55pw_f1efbed5-7c8e-44bf-8f15-ac8cd1dc205b/kube-multus/0.log" Jul 23 01:06:49.926303 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.926263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4w22f_ac42cc77-e8f3-4ee1-a85b-03b0750efc61/network-metrics-daemon/0.log" Jul 23 01:06:49.945909 ip-10-0-139-142 kubenswrapper[2585]: I0723 01:06:49.945883 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4w22f_ac42cc77-e8f3-4ee1-a85b-03b0750efc61/kube-rbac-proxy/0.log"