Jul 29 07:14:50.100424 ip-10-0-136-73 systemd[1]: Starting Kubernetes Kubelet... Jul 29 07:14:50.499889 ip-10-0-136-73 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 07:14:50.499889 ip-10-0-136-73 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 07:14:50.499889 ip-10-0-136-73 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 07:14:50.499889 ip-10-0-136-73 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 07:14:50.499889 ip-10-0-136-73 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 07:14:50.500666 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.500570 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 07:14:50.505808 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505776 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:14:50.505808 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505802 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:14:50.505808 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505806 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:14:50.505808 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505809 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:14:50.505808 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505812 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505815 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505818 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505821 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505823 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505826 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505829 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505831 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505834 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505836 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505839 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505841 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505844 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505848 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505851 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505854 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505857 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505859 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505862 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505864 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:14:50.505981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505875 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505880 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505883 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505886 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505888 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505891 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505894 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505896 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505899 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505901 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505906 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505910 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505913 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505915 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505920 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505923 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505926 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505929 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505932 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:14:50.506487 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505934 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505936 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505939 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505941 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505944 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505946 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505948 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505951 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505953 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505955 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505958 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505960 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505963 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505965 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505968 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505971 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505973 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505976 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505978 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505981 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:14:50.506941 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505983 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505986 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505988 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505991 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505996 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.505999 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506002 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506006 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506010 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506013 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506015 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506018 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506020 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506023 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506025 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506028 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506030 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506033 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506035 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:14:50.507523 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506052 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506055 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506059 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506062 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506076 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506079 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506083 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506087 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506090 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506093 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506095 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506098 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506100 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506103 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506105 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506108 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506111 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506860 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506867 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:14:50.507990 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506870 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506874 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506879 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506883 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506886 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506890 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506894 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506898 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506900 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506903 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506905 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506908 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506911 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506913 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506915 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506918 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506920 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506924 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506926 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:14:50.508478 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506929 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506931 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506934 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506936 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506939 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506941 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506944 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506948 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506950 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506953 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506955 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506958 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506960 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506963 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506966 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506969 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506972 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506975 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506977 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506980 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:14:50.508955 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506982 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506985 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506988 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506990 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506993 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.506995 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507000 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507002 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507005 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507007 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507010 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507013 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507016 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507018 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507020 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507023 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507025 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507028 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507030 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507033 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:14:50.509474 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507035 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507054 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507056 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507059 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507061 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507064 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507066 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507069 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507071 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507074 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507077 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507080 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507083 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507086 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507088 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507091 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507094 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507097 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507100 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507103 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:14:50.509970 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507105 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507108 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507110 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507113 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507116 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507119 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507121 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507124 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507127 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507129 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507132 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507134 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507137 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507139 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507141 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507144 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507146 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507149 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507530 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507535 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:14:50.510501 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507538 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507541 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507545 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507548 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507550 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507553 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507556 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507558 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507561 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507563 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507566 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507569 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507571 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507574 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507576 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507579 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507582 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507584 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507587 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507589 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:14:50.510997 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507594 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507597 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507600 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507603 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507606 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507609 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507612 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507615 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507617 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507620 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507622 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507625 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507628 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507630 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507633 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507640 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507643 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507645 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507648 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507650 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:14:50.511496 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507653 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507655 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507658 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507660 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507663 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507665 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507670 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507673 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507676 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507678 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507680 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507683 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507686 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507688 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507691 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507694 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507697 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507699 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507702 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507705 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:14:50.511981 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507707 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507709 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507713 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507715 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507718 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507722 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507726 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507729 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507732 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507735 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507738 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507740 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507743 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507745 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507748 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507751 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507754 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507756 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507760 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507763 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:14:50.512498 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507765 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507767 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507771 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507774 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507776 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507779 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507782 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507784 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507787 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507789 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507792 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507794 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507797 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507799 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507801 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507804 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.507806 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507897 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507908 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507916 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 07:14:50.512987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507923 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507934 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507940 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507945 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507949 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507952 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507955 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507958 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507962 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507965 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507970 2568 flags.go:64] FLAG: --cgroup-root="" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507973 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507976 2568 flags.go:64] FLAG: --client-ca-file="" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507979 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507983 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507988 2568 flags.go:64] FLAG: --cluster-domain="" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507991 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507994 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.507998 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508001 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508005 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508009 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508012 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 07:14:50.513507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508015 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508018 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508021 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508024 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508027 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508030 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508034 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508052 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508055 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508058 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508061 2568 flags.go:64] FLAG: --enable-server="true" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508064 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508069 2568 flags.go:64] FLAG: --event-burst="100" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508072 2568 flags.go:64] FLAG: --event-qps="50" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508075 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508078 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508080 2568 flags.go:64] FLAG: --eviction-hard="" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508086 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508089 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508093 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508096 2568 flags.go:64] FLAG: --eviction-soft="" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508099 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508102 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508106 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508108 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 07:14:50.514094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508111 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508114 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508117 2568 flags.go:64] FLAG: --feature-gates="" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508120 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508123 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508126 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508129 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508132 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508135 2568 flags.go:64] FLAG: --help="false" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508138 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508141 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508144 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508146 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508150 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508153 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508156 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508159 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508161 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508164 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508167 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508170 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508173 2568 flags.go:64] FLAG: --kube-reserved="" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508176 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508179 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 07:14:50.514764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508182 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508184 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508189 2568 flags.go:64] FLAG: --lock-file="" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508192 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508195 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508197 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508203 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508206 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508209 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508212 2568 flags.go:64] FLAG: --logging-format="text" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508215 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508218 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508221 2568 flags.go:64] FLAG: --manifest-url="" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508224 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508228 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508231 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508235 2568 flags.go:64] FLAG: --max-pods="110" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508238 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508241 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508244 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508247 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508250 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508253 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508256 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508264 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 07:14:50.515380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508267 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508269 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508272 2568 flags.go:64] FLAG: --pod-cidr="" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508276 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508282 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508285 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508288 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508291 2568 flags.go:64] FLAG: --port="10250" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508294 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508298 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ccf496ceffd53b6e" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508301 2568 flags.go:64] FLAG: --qos-reserved="" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508304 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508307 2568 flags.go:64] FLAG: --register-node="true" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508310 2568 flags.go:64] FLAG: --register-with-taints="" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508314 2568 flags.go:64] FLAG: --registry-burst="10" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508317 2568 flags.go:64] FLAG: --registry-qps="5" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508319 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508322 2568 flags.go:64] FLAG: --reserved-memory="" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508326 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508329 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508332 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508334 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508337 2568 flags.go:64] FLAG: --runonce="false" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508340 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508343 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 07:14:50.515987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508346 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508349 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508351 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508354 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508357 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508360 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508363 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508365 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508368 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508371 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508374 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508377 2568 flags.go:64] FLAG: --system-cgroups="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508379 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508385 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508388 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508391 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508396 2568 flags.go:64] FLAG: --tls-min-version="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508399 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508401 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508404 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508407 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508410 2568 flags.go:64] FLAG: --v="2" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508414 2568 flags.go:64] FLAG: --version="false" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508418 2568 flags.go:64] FLAG: --vmodule="" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508422 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 07:14:50.516623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.508425 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508544 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508548 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508551 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508554 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508557 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508560 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508562 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508564 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508567 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508570 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508572 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508574 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508577 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508580 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508582 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508585 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508588 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508590 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508593 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:14:50.517290 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508596 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508598 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508600 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508604 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508606 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508609 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508611 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508613 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508617 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508620 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508622 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508624 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508627 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508629 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508631 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508634 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508636 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508639 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508643 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508646 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:14:50.517783 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508648 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508651 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508653 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508656 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508658 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508661 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508664 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508666 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508669 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508671 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508673 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508676 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508680 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508682 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508684 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508687 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508689 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508692 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508694 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:14:50.518320 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508697 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508702 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508706 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508710 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508713 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508716 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508720 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508724 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508727 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508729 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508732 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508735 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508737 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508740 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508742 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508745 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508747 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508750 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508752 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:14:50.518794 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508755 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508757 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508760 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508762 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508765 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508767 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508769 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508772 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508775 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508778 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508780 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508783 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508786 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508788 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508792 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508796 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508799 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508802 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508805 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508808 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:14:50.519491 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508811 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.508813 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.509310 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.516266 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.516286 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.516305 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.516315 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516383 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516390 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516394 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516399 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516402 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516406 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516409 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:14:50.520325 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516413 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516415 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516419 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516422 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516425 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516429 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516432 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516435 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516438 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516440 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516443 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516446 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516449 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516452 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516455 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516457 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516461 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516463 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516466 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516469 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:14:50.520679 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516472 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516474 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516478 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516481 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516484 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516486 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516489 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516492 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516494 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516496 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516499 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516502 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516504 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516507 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516509 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516512 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516514 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516517 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516520 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516522 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:14:50.521188 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516525 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516527 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516529 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516533 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516535 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516537 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516540 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516542 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516545 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516547 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516549 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516552 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516554 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516557 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516560 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516563 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516565 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516568 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516570 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516573 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:14:50.521686 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516576 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516578 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516581 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516583 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516586 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516588 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516591 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516593 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516596 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516598 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516601 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516604 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516607 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516609 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516611 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516614 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516616 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516618 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516621 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:14:50.522250 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516623 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516627 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516631 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516634 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516637 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516639 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516643 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516646 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516648 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516651 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516654 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516656 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516659 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.516664 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516784 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:14:50.522721 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516789 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516792 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516795 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516799 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516803 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516805 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516808 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516811 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516813 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516816 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516820 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516823 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516825 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516828 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516830 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516833 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516835 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516838 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516840 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516843 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:14:50.523123 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516845 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516848 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516850 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516852 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516855 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516858 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516861 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516863 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516866 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516868 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516871 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516873 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516876 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516878 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516881 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516883 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516885 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516888 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516890 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516893 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:14:50.523608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516895 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516898 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516900 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516903 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516905 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516908 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516910 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516913 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516915 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516918 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516920 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516922 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516925 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516927 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516930 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516932 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516935 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516938 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516941 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516945 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:14:50.524113 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516948 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516951 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516954 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516957 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516959 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516962 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516965 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516967 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516970 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516972 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516974 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516977 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516979 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516982 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516984 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516986 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516989 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516992 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516994 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516997 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:14:50.524613 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.516999 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517001 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517004 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517006 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517008 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517011 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517013 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517017 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517021 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517024 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517027 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517030 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517033 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517035 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517055 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517058 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517061 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:14:50.525164 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:50.517064 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.517069 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.517208 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.519565 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.521160 2568 server.go:1019] "Starting client certificate rotation" Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.521271 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.521312 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.523441 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 07:14:50.525568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.524613 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 07:14:50.538098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.538067 2568 log.go:25] "Validated CRI v1 runtime API" Jul 29 07:14:50.540236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.540217 2568 log.go:25] "Validated CRI v1 image API" Jul 29 07:14:50.543575 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.543555 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 07:14:50.546304 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.546280 2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9045218c-4935-4b30-aaaa-c079a1e22d4f:/dev/nvme0n1p3 ed916784-8901-4c0f-91e6-1cfeb1158d6b:/dev/nvme0n1p4] Jul 29 07:14:50.546396 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.546303 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 07:14:50.551510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.551377 2568 manager.go:216] Machine: {Timestamp:2026-07-29 07:14:50.550224568 +0000 UTC m=+0.321675360 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111595 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b92d88ea7acca641a5280ae781cd SystemUUID:ec24b92d-88ea-7acc-a641-a5280ae781cd BootID:d16075c3-3fcf-46cb-bdfb-8e3dba2de11f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:88:fd:4a:e6:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:88:fd:4a:e6:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:46:df:9b:a6:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 07:14:50.551510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.551494 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 07:14:50.551666 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.551592 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 07:14:50.553847 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.553827 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 07:14:50.554269 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.554233 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 07:14:50.554427 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.554275 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 07:14:50.554939 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.554928 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 07:14:50.554976 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.554941 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 07:14:50.554976 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.554949 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 07:14:50.555452 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.555440 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 07:14:50.556634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.556621 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 07:14:50.559318 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559302 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 29 07:14:50.559471 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559460 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 07:14:50.559517 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559497 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 29 07:14:50.559517 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559507 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 07:14:50.559580 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559522 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 07:14:50.559580 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559532 2568 kubelet.go:398] "Adding apiserver pod source" Jul 29 07:14:50.559580 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559542 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 07:14:50.559961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559949 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 07:14:50.560008 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.559971 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 07:14:50.562890 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.562866 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 07:14:50.565599 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.565554 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 07:14:50.565717 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.565626 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 07:14:50.567051 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567016 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 07:14:50.567051 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567058 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567066 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567072 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567081 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567086 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567095 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567100 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567107 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567113 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567127 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 07:14:50.567172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567132 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 07:14:50.567830 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567810 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 07:14:50.567830 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.567831 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 07:14:50.569548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.569525 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-73.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 07:14:50.570162 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.570138 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 07:14:50.570202 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.570138 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 07:14:50.571324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.571312 2568 server.go:1268] "Started kubelet" Jul 29 07:14:50.571411 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.571385 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 07:14:50.571514 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.571469 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 07:14:50.571576 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.571541 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 07:14:50.572526 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.572508 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 07:14:50.572635 ip-10-0-136-73 systemd[1]: Started Kubernetes Kubelet. Jul 29 07:14:50.573314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.572734 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 29 07:14:50.576020 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.575997 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 07:14:50.577264 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.576534 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-73.ec2.internal.18c6b1573eb4ab9e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-73.ec2.internal,UID:ip-10-0-136-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-73.ec2.internal,},FirstTimestamp:2026-07-29 07:14:50.571279262 +0000 UTC m=+0.342730052,LastTimestamp:2026-07-29 07:14:50.571279262 +0000 UTC m=+0.342730052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-73.ec2.internal,}" Jul 29 07:14:50.578743 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.578720 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 07:14:50.578743 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.578736 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 07:14:50.578967 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.578947 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:50.579343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579324 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 07:14:50.579440 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579348 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 07:14:50.579440 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579361 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 07:14:50.579681 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579663 2568 factory.go:55] Registering systemd factory Jul 29 07:14:50.579681 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579683 2568 factory.go:223] Registration of the systemd container factory successfully Jul 29 07:14:50.579892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579877 2568 factory.go:153] Registering CRI-O factory Jul 29 07:14:50.579892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579892 2568 factory.go:223] Registration of the crio container factory successfully Jul 29 07:14:50.580069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579950 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 07:14:50.580069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579974 2568 factory.go:103] Registering Raw factory Jul 29 07:14:50.580069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.579988 2568 manager.go:1196] Started watching for new ooms in manager Jul 29 07:14:50.580437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.580422 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 29 07:14:50.580437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.580434 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 29 07:14:50.580712 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.580698 2568 manager.go:349] Starting recovery of all containers Jul 29 07:14:50.581912 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.581323 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 07:14:50.581912 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.581548 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 07:14:50.591568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.591427 2568 manager.go:354] Recovery completed Jul 29 07:14:50.592961 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.592938 2568 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 29 07:14:50.597317 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.597292 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.600238 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.600207 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.600363 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.600255 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.600363 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.600273 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.601176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601146 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 07:14:50.601176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601161 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 07:14:50.601176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601179 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 29 07:14:50.601342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601153 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.601960 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601945 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.602024 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601972 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.602024 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.601982 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.602680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.602661 2568 policy_none.go:49] "None policy: Start" Jul 29 07:14:50.602680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.602678 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 07:14:50.602773 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.602688 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 07:14:50.603631 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.603620 2568 policy_none.go:47] "Start" Jul 29 07:14:50.604970 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.604953 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csd2b" Jul 29 07:14:50.608975 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.608896 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-73.ec2.internal.18c6b157406e8060 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-73.ec2.internal,UID:ip-10-0-136-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-73.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-73.ec2.internal,},FirstTimestamp:2026-07-29 07:14:50.600235104 +0000 UTC m=+0.371685901,LastTimestamp:2026-07-29 07:14:50.600235104 +0000 UTC m=+0.371685901,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-73.ec2.internal,}" Jul 29 07:14:50.612166 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.612078 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-73.ec2.internal.18c6b157406eeacc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-73.ec2.internal,UID:ip-10-0-136-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-73.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-73.ec2.internal,},FirstTimestamp:2026-07-29 07:14:50.600262348 +0000 UTC m=+0.371713139,LastTimestamp:2026-07-29 07:14:50.600262348 +0000 UTC m=+0.371713139,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-73.ec2.internal,}" Jul 29 07:14:50.613235 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.613193 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csd2b" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642211 2568 manager.go:337] "Starting Device Plugin manager" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.642274 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642291 2568 server.go:87] "Starting device plugin registration server" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642606 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642620 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642751 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642844 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.642854 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.643405 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 07:14:50.653348 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.643466 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:50.729577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.729516 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 07:14:50.731016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.730993 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 07:14:50.731158 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.731052 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 07:14:50.731158 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.731083 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 07:14:50.731158 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.731132 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 07:14:50.733482 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.733333 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:14:50.743533 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.743492 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.744720 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.744697 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.744854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.744737 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.744854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.744748 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.744854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.744770 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.754596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.754525 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.754596 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.754559 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-73.ec2.internal\": node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:50.779095 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.779069 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:50.832035 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.831970 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal"] Jul 29 07:14:50.832226 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.832090 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.833478 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.833460 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.833572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.833502 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.833572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.833515 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.834809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.834794 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.834917 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.834900 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.834968 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.834933 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.835622 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.835604 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.835689 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.835635 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.835689 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.835650 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.835755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.835699 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.835755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.835728 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.835755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.835741 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.836801 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.836787 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.836841 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.836813 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:50.837550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.837537 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:50.837604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.837563 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:50.837604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.837573 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:50.859897 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.859863 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-73.ec2.internal\" not found" node="ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.864345 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.864326 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-73.ec2.internal\" not found" node="ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.879775 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.879746 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:50.881992 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.881968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f4c35dc2bc37a41db0b1d7d69155e3a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal\" (UID: \"1f4c35dc2bc37a41db0b1d7d69155e3a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.882100 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.882007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f4c35dc2bc37a41db0b1d7d69155e3a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal\" (UID: \"1f4c35dc2bc37a41db0b1d7d69155e3a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.882100 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.882035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1111a70e9f15469ee95cf515ced571c7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-73.ec2.internal\" (UID: \"1111a70e9f15469ee95cf515ced571c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.980191 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:50.980158 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:50.982395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.982370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1111a70e9f15469ee95cf515ced571c7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-73.ec2.internal\" (UID: \"1111a70e9f15469ee95cf515ced571c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.982497 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.982407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f4c35dc2bc37a41db0b1d7d69155e3a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal\" (UID: \"1f4c35dc2bc37a41db0b1d7d69155e3a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.982497 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.982431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f4c35dc2bc37a41db0b1d7d69155e3a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal\" (UID: \"1f4c35dc2bc37a41db0b1d7d69155e3a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.982497 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.982486 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1111a70e9f15469ee95cf515ced571c7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-73.ec2.internal\" (UID: \"1111a70e9f15469ee95cf515ced571c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.982596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.982487 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f4c35dc2bc37a41db0b1d7d69155e3a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal\" (UID: \"1f4c35dc2bc37a41db0b1d7d69155e3a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:50.982596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:50.982486 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f4c35dc2bc37a41db0b1d7d69155e3a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal\" (UID: \"1f4c35dc2bc37a41db0b1d7d69155e3a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:51.080935 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.080848 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.162262 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.162222 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:51.167946 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.167922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" Jul 29 07:14:51.181477 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.181445 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.282019 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.281978 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.382618 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.382549 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.483119 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.483085 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.520419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.520396 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 07:14:51.520961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.520571 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 07:14:51.545174 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.545146 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:14:51.583187 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.583147 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.617356 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.617310 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 07:09:51 +0000 UTC" deadline="2027-12-25 16:11:01.829934955 +0000 UTC" Jul 29 07:14:51.617493 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.617357 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12344h56m10.212589789s" Jul 29 07:14:51.640586 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.640496 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:14:51.643549 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.643525 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:14:51.643669 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.643561 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:14:51.643669 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.643573 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:14:51.683613 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.683574 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.762247 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:51.762212 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4c35dc2bc37a41db0b1d7d69155e3a.slice/crio-a23119337f1ec4f209ecb990f4fe03ce501607f62e7d8d29f7cebd1e979f6a02 WatchSource:0}: Error finding container a23119337f1ec4f209ecb990f4fe03ce501607f62e7d8d29f7cebd1e979f6a02: Status 404 returned error can't find the container with id a23119337f1ec4f209ecb990f4fe03ce501607f62e7d8d29f7cebd1e979f6a02 Jul 29 07:14:51.767893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.767874 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:14:51.783891 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.783850 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.787778 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:51.787748 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1111a70e9f15469ee95cf515ced571c7.slice/crio-4bd722db74ac90806549645545e4002b8a11326ec24a4adb0c8cff16dd627e5c WatchSource:0}: Error finding container 4bd722db74ac90806549645545e4002b8a11326ec24a4adb0c8cff16dd627e5c: Status 404 returned error can't find the container with id 4bd722db74ac90806549645545e4002b8a11326ec24a4adb0c8cff16dd627e5c Jul 29 07:14:51.826643 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.826611 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:14:51.884107 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:51.884074 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-73.ec2.internal\" not found" Jul 29 07:14:51.971180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.971115 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:14:51.979379 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.979346 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" Jul 29 07:14:51.987149 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.987120 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 07:14:51.988122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.988104 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" Jul 29 07:14:51.999885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:51.999857 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 07:14:52.561288 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.561246 2568 apiserver.go:52] "Watching apiserver" Jul 29 07:14:52.570643 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.570608 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 07:14:52.572838 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.572807 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-82z89","kube-system/konnectivity-agent-6wm6z","openshift-cluster-node-tuning-operator/tuned-r975p","openshift-dns/node-resolver-z22p2","openshift-image-registry/node-ca-8zxgt","openshift-multus/multus-additional-cni-plugins-mj742","openshift-multus/multus-z4c6q","openshift-network-operator/iptables-alerter-nljc8","kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal","openshift-multus/network-metrics-daemon-df8cd","openshift-network-diagnostics/network-check-target-tgb4r","openshift-ovn-kubernetes/ovnkube-node-lrthw"] Jul 29 07:14:52.574826 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.574797 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.576340 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.576281 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:52.576478 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.576383 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:14:52.577603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.577446 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.577740 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.577728 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 07:14:52.577962 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.577944 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 07:14:52.578014 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.577978 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m7sks\"" Jul 29 07:14:52.578141 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.578127 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.578239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.578223 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.578838 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.578791 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.579080 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.579058 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 07:14:52.580062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.580043 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7q5d2\"" Jul 29 07:14:52.581156 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.580356 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.581156 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.580574 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.581156 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.580783 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.581714 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.581506 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.581864 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.581836 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxhbp\"" Jul 29 07:14:52.582157 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.582137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.584229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.583639 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 07:14:52.584229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.583705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.584229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.583777 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.584756 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.584731 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.585009 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.584987 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pzv6s\"" Jul 29 07:14:52.585223 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.585202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.585306 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.585267 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.585306 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.585283 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:14:52.585693 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.585673 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 07:14:52.585964 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.585948 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fsr7d\"" Jul 29 07:14:52.586317 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.586292 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 07:14:52.587319 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.587285 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.587647 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.587629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.588663 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.588638 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.588934 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.588913 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 07:14:52.589041 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.588946 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 07:14:52.589280 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.589257 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fwv4d\"" Jul 29 07:14:52.589524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.589507 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 07:14:52.589674 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.589660 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 07:14:52.590476 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.590454 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.590822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.590802 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x7crp\"" Jul 29 07:14:52.591070 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591050 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 07:14:52.591279 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-tuned\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.591365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9045afd8-91a2-4b61-8424-19c537bd41ee-serviceca\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.591365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-system-cni-dir\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.591365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591062 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 07:14:52.591512 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmxx\" (UniqueName: \"kubernetes.io/projected/087ed21b-a53d-48ce-9f53-7894005159c3-kube-api-access-xkmxx\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.591512 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d61f9df2-2b99-4b3e-9ceb-768569ced90d-cni-binary-copy\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.591512 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591427 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-k8s-cni-cncf-io\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.591512 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591464 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:52.591512 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-daemon-config\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.591707 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591526 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nt57\" (UniqueName: \"kubernetes.io/projected/d61f9df2-2b99-4b3e-9ceb-768569ced90d-kube-api-access-9nt57\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.591707 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.591528 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:14:52.591804 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591723 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 07:14:52.591804 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.591780 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.592070 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592049 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9045afd8-91a2-4b61-8424-19c537bd41ee-host\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.592180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv8lk\" (UniqueName: \"kubernetes.io/projected/9045afd8-91a2-4b61-8424-19c537bd41ee-kube-api-access-fv8lk\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.592180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-system-cni-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-systemd\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-run\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-tmp-dir\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.592347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-hostroot\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnx9h\" (UniqueName: \"kubernetes.io/projected/65ab453e-08e8-4996-bd8e-a9663fceea63-kube-api-access-vnx9h\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592339 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592359 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-cnibin\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-os-release\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592442 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysconfig\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-kubernetes\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysctl-conf\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592528 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-socket-dir-parent\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-netns\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-modprobe-d\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-sys\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-lib-modules\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592695 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ab453e-08e8-4996-bd8e-a9663fceea63-tmp\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-conf-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592746 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-multus-certs\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-hosts-file\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592784 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtg4\" (UniqueName: \"kubernetes.io/projected/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-kube-api-access-kwtg4\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-cnibin\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-os-release\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592904 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-kubelet\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.592998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.592968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-cni-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-etc-kubernetes\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysctl-d\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593093 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6984b\"" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-cni-bin\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-cni-multus\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-var-lib-kubelet\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-host\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.593664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593468 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.594132 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593678 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 07:14:52.594132 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.593979 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.594451 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.594431 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fq6fw\"" Jul 29 07:14:52.594451 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.594443 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:14:52.594882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.594862 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 07:14:52.594942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.594866 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 07:14:52.612526 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.612480 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p6p4l" Jul 29 07:14:52.621550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.621514 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p6p4l" Jul 29 07:14:52.680483 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.680451 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 07:14:52.694365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-cni-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.694365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-etc-kubernetes\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysctl-d\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-slash\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694479 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db0797f0-49fe-4928-9446-e4ea9189531c-host-slash\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-cni-bin\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-cni-multus\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-var-lib-kubelet\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-host\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-system-cni-dir\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.694618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694633 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-cni-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694648 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-kubelet\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovnkube-config\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nt57\" (UniqueName: \"kubernetes.io/projected/d61f9df2-2b99-4b3e-9ceb-768569ced90d-kube-api-access-9nt57\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9045afd8-91a2-4b61-8424-19c537bd41ee-host\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694729 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-system-cni-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-run\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694800 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-hostroot\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnx9h\" (UniqueName: \"kubernetes.io/projected/65ab453e-08e8-4996-bd8e-a9663fceea63-kube-api-access-vnx9h\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-etc-selinux\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694893 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysconfig\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysctl-conf\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694956 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-etc-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.694973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-socket-dir-parent\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-netns\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-etc-kubernetes\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ab453e-08e8-4996-bd8e-a9663fceea63-tmp\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtg4\" (UniqueName: \"kubernetes.io/projected/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-kube-api-access-kwtg4\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-cni-bin\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysctl-d\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-env-overrides\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695219 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovn-node-metrics-cert\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-device-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-conf-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-multus-certs\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695300 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40135fd0-f3bb-40d5-986b-9530c70706fa-agent-certs\") pod \"konnectivity-agent-6wm6z\" (UID: \"40135fd0-f3bb-40d5-986b-9530c70706fa\") " pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695332 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfjx\" (UniqueName: \"kubernetes.io/projected/e417ca26-6f12-4ddc-a68a-1c4895d1f881-kube-api-access-sqfjx\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695354 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db0797f0-49fe-4928-9446-e4ea9189531c-iptables-alerter-script\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695411 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-socket-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.695902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-tuned\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9045afd8-91a2-4b61-8424-19c537bd41ee-serviceca\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmxx\" (UniqueName: \"kubernetes.io/projected/087ed21b-a53d-48ce-9f53-7894005159c3-kube-api-access-xkmxx\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695499 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-netns\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-cni-netd\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-sys-fs\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d61f9df2-2b99-4b3e-9ceb-768569ced90d-cni-binary-copy\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695603 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-k8s-cni-cncf-io\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-daemon-config\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8lk\" (UniqueName: \"kubernetes.io/projected/9045afd8-91a2-4b61-8424-19c537bd41ee-kube-api-access-fv8lk\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695682 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c71bd361-1cda-4ccd-8daf-3fc786c6d773-kubelet-config\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40135fd0-f3bb-40d5-986b-9530c70706fa-konnectivity-ca\") pod \"konnectivity-agent-6wm6z\" (UID: \"40135fd0-f3bb-40d5-986b-9530c70706fa\") " pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-systemd-units\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695774 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.695808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-k8s-cni-cncf-io\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696204 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d61f9df2-2b99-4b3e-9ceb-768569ced90d-cni-binary-copy\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-cni-bin\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.696614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysconfig\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-cni-multus\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-daemon-config\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-system-cni-dir\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696388 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-sysctl-conf\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696398 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-system-cni-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696417 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-var-lib-kubelet\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696440 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-systemd\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696470 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-hostroot\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-tmp-dir\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696482 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-socket-dir-parent\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-run-netns\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696533 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-registration-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696547 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-systemd\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696444 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-run\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696587 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9045afd8-91a2-4b61-8424-19c537bd41ee-host\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696607 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-multus-conf-dir\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.697500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4kw\" (UniqueName: \"kubernetes.io/projected/db0797f0-49fe-4928-9446-e4ea9189531c-kube-api-access-2t4kw\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696625 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-run-multus-certs\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwqw\" (UniqueName: \"kubernetes.io/projected/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-kube-api-access-mlwqw\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696744 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-host\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-cnibin\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696829 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-tmp-dir\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696871 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-os-release\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696902 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-cnibin\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-kubernetes\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.696998 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-kubernetes\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c71bd361-1cda-4ccd-8daf-3fc786c6d773-dbus\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-systemd\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697112 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-os-release\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697160 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvmj\" (UniqueName: \"kubernetes.io/projected/24546c0d-8683-4aac-8dd8-2f077292bc4f-kube-api-access-pmvmj\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-modprobe-d\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-sys\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.698378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697312 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9045afd8-91a2-4b61-8424-19c537bd41ee-serviceca\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697331 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-lib-modules\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697373 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-sys\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-var-lib-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697453 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-node-log\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-log-socket\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-hosts-file\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697543 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-modprobe-d\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697615 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65ab453e-08e8-4996-bd8e-a9663fceea63-lib-modules\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-cnibin\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-os-release\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697674 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-hosts-file\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-cnibin\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/087ed21b-a53d-48ce-9f53-7894005159c3-os-release\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-ovn\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.699016 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.699755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovnkube-script-lib\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.699755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.699755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.697968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-kubelet\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.699755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.698016 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d61f9df2-2b99-4b3e-9ceb-768569ced90d-host-var-lib-kubelet\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.699755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.698330 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/087ed21b-a53d-48ce-9f53-7894005159c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.699985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.699856 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ab453e-08e8-4996-bd8e-a9663fceea63-tmp\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.699985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.699886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65ab453e-08e8-4996-bd8e-a9663fceea63-etc-tuned\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.706059 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.705982 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:14:52.706247 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.706051 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:14:52.706247 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.706087 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:52.706247 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.706165 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:14:53.206139922 +0000 UTC m=+2.977590716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:52.710443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.710409 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nt57\" (UniqueName: \"kubernetes.io/projected/d61f9df2-2b99-4b3e-9ceb-768569ced90d-kube-api-access-9nt57\") pod \"multus-z4c6q\" (UID: \"d61f9df2-2b99-4b3e-9ceb-768569ced90d\") " pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.710614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.710412 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8lk\" (UniqueName: \"kubernetes.io/projected/9045afd8-91a2-4b61-8424-19c537bd41ee-kube-api-access-fv8lk\") pod \"node-ca-8zxgt\" (UID: \"9045afd8-91a2-4b61-8424-19c537bd41ee\") " pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.711273 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.711247 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmxx\" (UniqueName: \"kubernetes.io/projected/087ed21b-a53d-48ce-9f53-7894005159c3-kube-api-access-xkmxx\") pod \"multus-additional-cni-plugins-mj742\" (UID: \"087ed21b-a53d-48ce-9f53-7894005159c3\") " pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.711428 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.711411 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnx9h\" (UniqueName: \"kubernetes.io/projected/65ab453e-08e8-4996-bd8e-a9663fceea63-kube-api-access-vnx9h\") pod \"tuned-r975p\" (UID: \"65ab453e-08e8-4996-bd8e-a9663fceea63\") " pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.712094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.712050 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtg4\" (UniqueName: \"kubernetes.io/projected/35d5b11f-9aca-4a3d-a010-c9cde5be22d4-kube-api-access-kwtg4\") pod \"node-resolver-z22p2\" (UID: \"35d5b11f-9aca-4a3d-a010-c9cde5be22d4\") " pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.736258 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.736189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" event={"ID":"1111a70e9f15469ee95cf515ced571c7","Type":"ContainerStarted","Data":"4bd722db74ac90806549645545e4002b8a11326ec24a4adb0c8cff16dd627e5c"} Jul 29 07:14:52.737396 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.737339 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" event={"ID":"1f4c35dc2bc37a41db0b1d7d69155e3a","Type":"ContainerStarted","Data":"a23119337f1ec4f209ecb990f4fe03ce501607f62e7d8d29f7cebd1e979f6a02"} Jul 29 07:14:52.799191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwqw\" (UniqueName: \"kubernetes.io/projected/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-kube-api-access-mlwqw\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.799191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c71bd361-1cda-4ccd-8daf-3fc786c6d773-dbus\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-systemd\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvmj\" (UniqueName: \"kubernetes.io/projected/24546c0d-8683-4aac-8dd8-2f077292bc4f-kube-api-access-pmvmj\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-var-lib-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799294 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-node-log\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-log-socket\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-ovn\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799326 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-systemd\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovnkube-script-lib\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799402 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-node-log\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-slash\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799422 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c71bd361-1cda-4ccd-8daf-3fc786c6d773-dbus\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.799468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db0797f0-49fe-4928-9446-e4ea9189531c-host-slash\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799490 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db0797f0-49fe-4928-9446-e4ea9189531c-host-slash\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-log-socket\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-kubelet\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799550 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-ovn\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799555 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovnkube-config\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799588 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799614 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-var-lib-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-etc-selinux\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.799671 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799682 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-etc-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799710 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-etc-selinux\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799722 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.799743 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:14:53.299719971 +0000 UTC m=+3.071170755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:52.800155 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799750 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-slash\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-cni-bin\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-run-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.799804 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-cni-bin\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-kubelet\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-etc-openvswitch\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.799930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-env-overrides\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:52.799984 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:14:53.299967866 +0000 UTC m=+3.071418644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800032 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovn-node-metrics-cert\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-device-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-env-overrides\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800509 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-device-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovnkube-script-lib\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800528 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovnkube-config\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800564 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40135fd0-f3bb-40d5-986b-9530c70706fa-agent-certs\") pod \"konnectivity-agent-6wm6z\" (UID: \"40135fd0-f3bb-40d5-986b-9530c70706fa\") " pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfjx\" (UniqueName: \"kubernetes.io/projected/e417ca26-6f12-4ddc-a68a-1c4895d1f881-kube-api-access-sqfjx\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.800925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db0797f0-49fe-4928-9446-e4ea9189531c-iptables-alerter-script\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800699 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-socket-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-cni-netd\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-sys-fs\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.800988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c71bd361-1cda-4ccd-8daf-3fc786c6d773-kubelet-config\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40135fd0-f3bb-40d5-986b-9530c70706fa-konnectivity-ca\") pod \"konnectivity-agent-6wm6z\" (UID: \"40135fd0-f3bb-40d5-986b-9530c70706fa\") " pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-systemd-units\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-run-netns\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801063 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-socket-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-run-netns\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c71bd361-1cda-4ccd-8daf-3fc786c6d773-kubelet-config\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-registration-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4kw\" (UniqueName: \"kubernetes.io/projected/db0797f0-49fe-4928-9446-e4ea9189531c-kube-api-access-2t4kw\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801231 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-sys-fs\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801274 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-host-cni-netd\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801315 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-registration-dir\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.802950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801182 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db0797f0-49fe-4928-9446-e4ea9189531c-iptables-alerter-script\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.802950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e417ca26-6f12-4ddc-a68a-1c4895d1f881-systemd-units\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.802950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.801619 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40135fd0-f3bb-40d5-986b-9530c70706fa-konnectivity-ca\") pod \"konnectivity-agent-6wm6z\" (UID: \"40135fd0-f3bb-40d5-986b-9530c70706fa\") " pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.802950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.802882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e417ca26-6f12-4ddc-a68a-1c4895d1f881-ovn-node-metrics-cert\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.803589 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.803567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40135fd0-f3bb-40d5-986b-9530c70706fa-agent-certs\") pod \"konnectivity-agent-6wm6z\" (UID: \"40135fd0-f3bb-40d5-986b-9530c70706fa\") " pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.812434 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.812353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvmj\" (UniqueName: \"kubernetes.io/projected/24546c0d-8683-4aac-8dd8-2f077292bc4f-kube-api-access-pmvmj\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:52.812568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.812528 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwqw\" (UniqueName: \"kubernetes.io/projected/e7e8d5f7-cd3a-4abb-8e02-359fb09c3575-kube-api-access-mlwqw\") pod \"aws-ebs-csi-driver-node-9wbh5\" (UID: \"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.813691 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.813666 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfjx\" (UniqueName: \"kubernetes.io/projected/e417ca26-6f12-4ddc-a68a-1c4895d1f881-kube-api-access-sqfjx\") pod \"ovnkube-node-lrthw\" (UID: \"e417ca26-6f12-4ddc-a68a-1c4895d1f881\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.813791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.813694 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4kw\" (UniqueName: \"kubernetes.io/projected/db0797f0-49fe-4928-9446-e4ea9189531c-kube-api-access-2t4kw\") pod \"iptables-alerter-nljc8\" (UID: \"db0797f0-49fe-4928-9446-e4ea9189531c\") " pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:52.895503 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.895464 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z4c6q" Jul 29 07:14:52.905216 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.905183 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r975p" Jul 29 07:14:52.916244 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.916204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" Jul 29 07:14:52.923310 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.923280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z22p2" Jul 29 07:14:52.930478 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.930452 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8zxgt" Jul 29 07:14:52.941336 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.941300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mj742" Jul 29 07:14:52.949341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.949308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:14:52.959326 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.959276 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:14:52.965490 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:52.965459 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nljc8" Jul 29 07:14:53.017379 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.017340 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:14:53.306191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.306096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:53.306191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.306166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.306198 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306319 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306332 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306354 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306361 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306384 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306385 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:14:54.306366248 +0000 UTC m=+4.077817027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:53.306453 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306446 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:14:54.306428062 +0000 UTC m=+4.077878846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:53.306866 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:53.306463 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:14:54.30645375 +0000 UTC m=+4.077904535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:53.624244 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.624143 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 07:09:53 +0000 UTC" deadline="2028-05-02 17:17:11.427374994 +0000 UTC" Jul 29 07:14:53.624244 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.624181 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15442h2m17.803197975s" Jul 29 07:14:53.719182 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.719149 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0797f0_49fe_4928_9446_e4ea9189531c.slice/crio-d4ed8acf33369df47f3c1e75afc658dd78f72f2343ef3e63bb873c4ec046dd71 WatchSource:0}: Error finding container d4ed8acf33369df47f3c1e75afc658dd78f72f2343ef3e63bb873c4ec046dd71: Status 404 returned error can't find the container with id d4ed8acf33369df47f3c1e75afc658dd78f72f2343ef3e63bb873c4ec046dd71 Jul 29 07:14:53.720589 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.720562 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d5b11f_9aca_4a3d_a010_c9cde5be22d4.slice/crio-c633b8b3228604175e61b6d2f696ddbee0b1c09b236a33a7592495be3e784109 WatchSource:0}: Error finding container c633b8b3228604175e61b6d2f696ddbee0b1c09b236a33a7592495be3e784109: Status 404 returned error can't find the container with id c633b8b3228604175e61b6d2f696ddbee0b1c09b236a33a7592495be3e784109 Jul 29 07:14:53.721307 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.721280 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087ed21b_a53d_48ce_9f53_7894005159c3.slice/crio-81390e4f9fd6149457d403a4f58432d4a7a9d7b8853d092673b8903dc99731b8 WatchSource:0}: Error finding container 81390e4f9fd6149457d403a4f58432d4a7a9d7b8853d092673b8903dc99731b8: Status 404 returned error can't find the container with id 81390e4f9fd6149457d403a4f58432d4a7a9d7b8853d092673b8903dc99731b8 Jul 29 07:14:53.723312 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.723250 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40135fd0_f3bb_40d5_986b_9530c70706fa.slice/crio-a8067aaa596cefd7da32fb9e1def39c2c4634e8695f90e6d9afb1382be3c6363 WatchSource:0}: Error finding container a8067aaa596cefd7da32fb9e1def39c2c4634e8695f90e6d9afb1382be3c6363: Status 404 returned error can't find the container with id a8067aaa596cefd7da32fb9e1def39c2c4634e8695f90e6d9afb1382be3c6363 Jul 29 07:14:53.725656 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.725563 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ab453e_08e8_4996_bd8e_a9663fceea63.slice/crio-bb0b63d4759123e08822c9a3288e4a83cbad48b1a19e17aa7943a6f1b1f60067 WatchSource:0}: Error finding container bb0b63d4759123e08822c9a3288e4a83cbad48b1a19e17aa7943a6f1b1f60067: Status 404 returned error can't find the container with id bb0b63d4759123e08822c9a3288e4a83cbad48b1a19e17aa7943a6f1b1f60067 Jul 29 07:14:53.727175 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.727145 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9045afd8_91a2_4b61_8424_19c537bd41ee.slice/crio-f451474ca4353aca76568acbd943aed3237314f5aa4fdbce82468a8af1e2e210 WatchSource:0}: Error finding container f451474ca4353aca76568acbd943aed3237314f5aa4fdbce82468a8af1e2e210: Status 404 returned error can't find the container with id f451474ca4353aca76568acbd943aed3237314f5aa4fdbce82468a8af1e2e210 Jul 29 07:14:53.728409 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.728374 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode417ca26_6f12_4ddc_a68a_1c4895d1f881.slice/crio-abe733b0dce2fa0d3ae21b18a2530f877ad952f6f402d08873f521d64e7457a5 WatchSource:0}: Error finding container abe733b0dce2fa0d3ae21b18a2530f877ad952f6f402d08873f521d64e7457a5: Status 404 returned error can't find the container with id abe733b0dce2fa0d3ae21b18a2530f877ad952f6f402d08873f521d64e7457a5 Jul 29 07:14:53.729155 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.729091 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e8d5f7_cd3a_4abb_8e02_359fb09c3575.slice/crio-dee7c358059f076419a13860ea44f2567fbe6521c44097a01f5d2e124d2d279e WatchSource:0}: Error finding container dee7c358059f076419a13860ea44f2567fbe6521c44097a01f5d2e124d2d279e: Status 404 returned error can't find the container with id dee7c358059f076419a13860ea44f2567fbe6521c44097a01f5d2e124d2d279e Jul 29 07:14:53.730098 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:14:53.730033 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61f9df2_2b99_4b3e_9ceb_768569ced90d.slice/crio-7ef72451cc6f0e0a801b12d0ed3c24cfdef75f1b284186843f3cb30cc277d180 WatchSource:0}: Error finding container 7ef72451cc6f0e0a801b12d0ed3c24cfdef75f1b284186843f3cb30cc277d180: Status 404 returned error can't find the container with id 7ef72451cc6f0e0a801b12d0ed3c24cfdef75f1b284186843f3cb30cc277d180 Jul 29 07:14:53.740135 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.740102 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"abe733b0dce2fa0d3ae21b18a2530f877ad952f6f402d08873f521d64e7457a5"} Jul 29 07:14:53.741841 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.741814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r975p" event={"ID":"65ab453e-08e8-4996-bd8e-a9663fceea63","Type":"ContainerStarted","Data":"bb0b63d4759123e08822c9a3288e4a83cbad48b1a19e17aa7943a6f1b1f60067"} Jul 29 07:14:53.743079 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.743052 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6wm6z" event={"ID":"40135fd0-f3bb-40d5-986b-9530c70706fa","Type":"ContainerStarted","Data":"a8067aaa596cefd7da32fb9e1def39c2c4634e8695f90e6d9afb1382be3c6363"} Jul 29 07:14:53.744447 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.744425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z22p2" event={"ID":"35d5b11f-9aca-4a3d-a010-c9cde5be22d4","Type":"ContainerStarted","Data":"c633b8b3228604175e61b6d2f696ddbee0b1c09b236a33a7592495be3e784109"} Jul 29 07:14:53.745794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.745761 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8zxgt" event={"ID":"9045afd8-91a2-4b61-8424-19c537bd41ee","Type":"ContainerStarted","Data":"f451474ca4353aca76568acbd943aed3237314f5aa4fdbce82468a8af1e2e210"} Jul 29 07:14:53.747639 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.747608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerStarted","Data":"81390e4f9fd6149457d403a4f58432d4a7a9d7b8853d092673b8903dc99731b8"} Jul 29 07:14:53.749499 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.749474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nljc8" event={"ID":"db0797f0-49fe-4928-9446-e4ea9189531c","Type":"ContainerStarted","Data":"d4ed8acf33369df47f3c1e75afc658dd78f72f2343ef3e63bb873c4ec046dd71"} Jul 29 07:14:53.750541 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.750519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z4c6q" event={"ID":"d61f9df2-2b99-4b3e-9ceb-768569ced90d","Type":"ContainerStarted","Data":"7ef72451cc6f0e0a801b12d0ed3c24cfdef75f1b284186843f3cb30cc277d180"} Jul 29 07:14:53.752140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:53.752080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" event={"ID":"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575","Type":"ContainerStarted","Data":"dee7c358059f076419a13860ea44f2567fbe6521c44097a01f5d2e124d2d279e"} Jul 29 07:14:54.314343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.314291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:54.314544 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.314355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:54.314544 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.314426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:54.314676 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.314593 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:14:54.314676 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.314614 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:14:54.314676 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.314628 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:54.314838 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.314717 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:14:56.314696305 +0000 UTC m=+6.086147106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:54.315218 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.315196 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:54.315316 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.315264 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:14:56.315245502 +0000 UTC m=+6.086696279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:54.315391 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.315339 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:54.315446 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.315400 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:14:56.315365883 +0000 UTC m=+6.086816668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:54.625376 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.625272 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 07:09:53 +0000 UTC" deadline="2028-02-05 11:31:22.660552028 +0000 UTC" Jul 29 07:14:54.625376 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.625319 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13348h16m28.035238367s" Jul 29 07:14:54.735932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.735897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:54.736178 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.736108 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:14:54.736293 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.736273 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:54.736403 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.736382 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:14:54.737043 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.737020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:54.737269 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:54.737248 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:14:54.796250 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.796158 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" event={"ID":"1111a70e9f15469ee95cf515ced571c7","Type":"ContainerStarted","Data":"2fe21c9af77ec610aaaa2ee35e0e3d7212538781e7375aeab9a497fe8ca8b9ee"} Jul 29 07:14:54.811636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.811590 2568 generic.go:356] "Generic (PLEG): container finished" podID="1f4c35dc2bc37a41db0b1d7d69155e3a" containerID="8311928934fb51558a1fefb7262720ced4e36cfc43ad37b8372fd731e8925bd5" exitCode=0 Jul 29 07:14:54.811843 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.811658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" event={"ID":"1f4c35dc2bc37a41db0b1d7d69155e3a","Type":"ContainerDied","Data":"8311928934fb51558a1fefb7262720ced4e36cfc43ad37b8372fd731e8925bd5"} Jul 29 07:14:54.813968 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:54.813592 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-73.ec2.internal" podStartSLOduration=2.813571071 podStartE2EDuration="2.813571071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:14:54.812647781 +0000 UTC m=+4.584098585" watchObservedRunningTime="2026-07-29 07:14:54.813571071 +0000 UTC m=+4.585021876" Jul 29 07:14:55.831238 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:55.831186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" event={"ID":"1f4c35dc2bc37a41db0b1d7d69155e3a","Type":"ContainerStarted","Data":"e98d3da6fe49f809a2973d423eb7523e076affa3b0d066f797064fcab3059544"} Jul 29 07:14:55.848370 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:55.848292 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-73.ec2.internal" podStartSLOduration=3.848267363 podStartE2EDuration="3.848267363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:14:55.847103603 +0000 UTC m=+5.618554399" watchObservedRunningTime="2026-07-29 07:14:55.848267363 +0000 UTC m=+5.619718168" Jul 29 07:14:56.338479 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:56.338437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:56.338675 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:56.338500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:56.338675 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:56.338583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:56.338675 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338652 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:56.338869 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338730 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:14:56.338869 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338757 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:14:56.338869 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338771 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:56.339009 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338871 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:56.339009 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338736 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:00.33871202 +0000 UTC m=+10.110162817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:14:56.339009 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338925 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:00.338904804 +0000 UTC m=+10.110355588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:14:56.339009 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.338944 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:15:00.338934124 +0000 UTC m=+10.110384903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:14:56.739688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:56.738710 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:56.739688 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.738860 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:14:56.739688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:56.739304 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:56.739688 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.739403 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:14:56.739688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:56.739465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:56.739688 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:56.739543 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:14:58.734073 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:58.733539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:14:58.734073 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:58.733691 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:14:58.734073 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:58.733851 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:14:58.734073 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:14:58.733851 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:14:58.734073 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:58.733994 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:14:58.734806 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:14:58.734135 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:00.383887 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:00.383843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:00.383925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:00.383976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384046 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384080 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384096 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384110 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384116 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384138 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:15:08.384116004 +0000 UTC m=+18.155566783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384158 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:08.384147708 +0000 UTC m=+18.155598491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:00.384421 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.384174 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:08.384165546 +0000 UTC m=+18.155616326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:00.733335 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:00.733250 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:00.733512 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.733388 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:00.733985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:00.733830 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:00.733985 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.733938 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:00.734457 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:00.734300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:00.734457 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:00.734419 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:02.732327 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:02.732279 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:02.732327 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:02.732318 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:02.732811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:02.732282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:02.732811 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:02.732435 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:02.732811 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:02.732536 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:02.732811 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:02.732601 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:04.731595 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:04.731554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:04.732207 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:04.731554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:04.732207 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:04.731683 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:04.732207 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:04.731557 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:04.732207 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:04.731764 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:04.732207 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:04.731850 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:06.731680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:06.731644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:06.731680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:06.731678 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:06.732198 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:06.731651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:06.732198 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:06.731787 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:06.732198 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:06.731877 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:06.732198 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:06.731980 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:08.444272 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:08.444226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:08.444272 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:08.444277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:08.444307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444415 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444433 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444459 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444473 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444474 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:15:24.444459684 +0000 UTC m=+34.215910461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444537 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:24.444523561 +0000 UTC m=+34.215974353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444582 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:08.444927 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.444605 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:24.444597829 +0000 UTC m=+34.216048638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:08.732240 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:08.732149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:08.732393 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:08.732161 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:08.732393 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.732266 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:08.732393 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:08.732283 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:08.732393 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.732352 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:08.732543 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:08.732439 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:10.732641 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:10.732601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:10.733114 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:10.732736 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:10.733114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:10.732753 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:10.733114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:10.732777 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:10.733114 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:10.732845 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:10.733114 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:10.732921 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:12.732130 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:12.732083 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:12.732687 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:12.732212 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:12.732687 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:12.732663 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:12.732806 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:12.732760 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:12.732920 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:12.732904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:12.733018 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:12.733002 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:13.875726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.875471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z4c6q" event={"ID":"d61f9df2-2b99-4b3e-9ceb-768569ced90d","Type":"ContainerStarted","Data":"ecbeff8bd17d287caf19c985d61b8191d438b39182584071978afcb449ca0d7a"} Jul 29 07:15:13.880170 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.880139 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" event={"ID":"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575","Type":"ContainerStarted","Data":"574f8c850da5ff1b29d321c487187cffee9ab31dc79dfcc4fee66c3b971c220a"} Jul 29 07:15:13.882300 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.882274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"2e7620ed7ffc829f1f892ab97de7905b12489ba81a3fcf8334bdfa459660f49c"} Jul 29 07:15:13.882427 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.882360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"b619d1903546f5391ad5ff750e4c7c498d8610ee397e7f658f8eb6731098ffdb"} Jul 29 07:15:13.882427 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.882422 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"80ac134ca3242067f75b88b4335f9b971eb9843d8c01aac095c1e1bf63f5be75"} Jul 29 07:15:13.882515 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.882468 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"42e07998cadc9c5ac50027a6106b8e7402048c17375b6cc87b67a86f2b14abb3"} Jul 29 07:15:13.883602 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.883581 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r975p" event={"ID":"65ab453e-08e8-4996-bd8e-a9663fceea63","Type":"ContainerStarted","Data":"f7e85144e4c6f6afb6288a4d06bf64ff99af72c0459d459c62f5025f0b00dc66"} Jul 29 07:15:13.884767 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.884744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6wm6z" event={"ID":"40135fd0-f3bb-40d5-986b-9530c70706fa","Type":"ContainerStarted","Data":"71711f90994a6d5cf2e91c5e03c7ad08440bd8b872c3bc9f738035e62d57e508"} Jul 29 07:15:13.886085 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.886062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z22p2" event={"ID":"35d5b11f-9aca-4a3d-a010-c9cde5be22d4","Type":"ContainerStarted","Data":"f0939636e8a43b3af8cfe65bf6fbb312201ec5a94fd2180aba1821b3e5f4babe"} Jul 29 07:15:13.887353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.887329 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8zxgt" event={"ID":"9045afd8-91a2-4b61-8424-19c537bd41ee","Type":"ContainerStarted","Data":"919c446242aca0875277bc2fdcbcb07e52f6c3463e1fac10b43caec4e77cea86"} Jul 29 07:15:13.888625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.888602 2568 generic.go:356] "Generic (PLEG): container finished" podID="087ed21b-a53d-48ce-9f53-7894005159c3" containerID="f48b9320b0d920a6815e0a49cb7329f8fa257d8a693e176c5d5a947403a8072d" exitCode=0 Jul 29 07:15:13.888705 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.888636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerDied","Data":"f48b9320b0d920a6815e0a49cb7329f8fa257d8a693e176c5d5a947403a8072d"} Jul 29 07:15:13.894368 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.894312 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z4c6q" podStartSLOduration=3.218769826 podStartE2EDuration="22.894296254s" totalImagesPullingTime="19.675526428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:13.89366443 +0000 UTC m=+23.665115223" watchObservedRunningTime="2026-07-29 07:15:13.894296254 +0000 UTC m=+23.665747053" Jul 29 07:15:13.909713 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.909664 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r975p" podStartSLOduration=3.281503308 podStartE2EDuration="22.909645294s" totalImagesPullingTime="19.628141986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:13.909117436 +0000 UTC m=+23.680568236" watchObservedRunningTime="2026-07-29 07:15:13.909645294 +0000 UTC m=+23.681096093" Jul 29 07:15:13.925895 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.925845 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z22p2" podStartSLOduration=3.295875078 podStartE2EDuration="22.925830081s" totalImagesPullingTime="19.629955003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:13.92561698 +0000 UTC m=+23.697067780" watchObservedRunningTime="2026-07-29 07:15:13.925830081 +0000 UTC m=+23.697280879" Jul 29 07:15:13.950228 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:13.950181 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8zxgt" podStartSLOduration=3.329278343 podStartE2EDuration="22.950165257s" totalImagesPullingTime="19.620886914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:13.949956488 +0000 UTC m=+23.721407289" watchObservedRunningTime="2026-07-29 07:15:13.950165257 +0000 UTC m=+23.721616056" Jul 29 07:15:14.732278 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.732236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:14.732526 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.732236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:14.732526 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:14.732353 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:14.732526 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:14.732458 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:14.732526 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.732235 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:14.732676 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:14.732568 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:14.893897 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.893850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nljc8" event={"ID":"db0797f0-49fe-4928-9446-e4ea9189531c","Type":"ContainerStarted","Data":"2ca3a46e5e1b10646b03b6b49bb880b46600d4bfc6436743af15b29cc54440fd"} Jul 29 07:15:14.898817 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.898787 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"f1893bb8680715b5d5ecc5efdc1d25d0db26eb89c0b1fdf377445efa408c7d82"} Jul 29 07:15:14.898975 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.898847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"4868bb3e499a3ee1b2dba3ec1a437f3db19aa8a51d76928163174057cba1ff22"} Jul 29 07:15:14.910284 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.910240 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6wm6z" podStartSLOduration=12.021149527 podStartE2EDuration="23.910220794s" totalImagesPullingTime="11.889071267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:13.997885796 +0000 UTC m=+23.769336595" watchObservedRunningTime="2026-07-29 07:15:14.910220794 +0000 UTC m=+24.681671592" Jul 29 07:15:14.910571 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.910551 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nljc8" podStartSLOduration=3.27889121 podStartE2EDuration="22.910544452s" totalImagesPullingTime="19.631653242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:14.90983146 +0000 UTC m=+24.681282262" watchObservedRunningTime="2026-07-29 07:15:14.910544452 +0000 UTC m=+24.681995250" Jul 29 07:15:14.979984 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:14.979943 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 07:15:15.655885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:15.655751 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T07:15:14.979966894Z","UUID":"0ebaa3f7-b592-44ba-ab3b-6faa02474ab0","Handler":null,"Name":"","Endpoint":""} Jul 29 07:15:15.657505 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:15.657482 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 07:15:15.657655 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:15.657526 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 07:15:15.902451 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:15.902414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" event={"ID":"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575","Type":"ContainerStarted","Data":"016982f72f209f5176d042140afa93dcfbaf534bdb3be2cfe06ab4ed9e80942b"} Jul 29 07:15:16.162834 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.162802 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:15:16.163634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.163600 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:15:16.732256 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.732154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:16.732424 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.732154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:16.732424 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:16.732295 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:16.732424 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:16.732401 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:16.732594 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.732464 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:16.732594 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:16.732538 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:16.907590 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.907548 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"b2d02a58d75a31ea92d71c9008b2529ed3e27590cf5cb20598c54c9f1782e80a"} Jul 29 07:15:16.909838 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.909806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" event={"ID":"e7e8d5f7-cd3a-4abb-8e02-359fb09c3575","Type":"ContainerStarted","Data":"5c7c48fc2725586233dd6e199e0f564093a645765709cf54568c6a40391f2627"} Jul 29 07:15:16.910007 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.909988 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:15:16.910539 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.910520 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6wm6z" Jul 29 07:15:16.929700 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:16.929643 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wbh5" podStartSLOduration=3.771809081 podStartE2EDuration="25.929626469s" totalImagesPullingTime="22.157817388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:16.92931598 +0000 UTC m=+26.700766779" watchObservedRunningTime="2026-07-29 07:15:16.929626469 +0000 UTC m=+26.701077268" Jul 29 07:15:18.731731 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:18.731528 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:18.732434 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:18.731528 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:18.732434 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:18.731811 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:18.732434 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:18.731901 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:18.732434 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:18.731528 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:18.732434 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:18.732009 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:18.918207 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:18.918166 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" event={"ID":"e417ca26-6f12-4ddc-a68a-1c4895d1f881","Type":"ContainerStarted","Data":"9df5618b85e5706300b3438bad990e5924b5a4ed324dbe306276bfbcdaed2f88"} Jul 29 07:15:18.963846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:18.963788 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" podStartSLOduration=8.249080197 podStartE2EDuration="27.963772756s" totalImagesPullingTime="19.714692559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:18.963101553 +0000 UTC m=+28.734552345" watchObservedRunningTime="2026-07-29 07:15:18.963772756 +0000 UTC m=+28.735223553" Jul 29 07:15:19.921210 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:19.921173 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:15:19.921648 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:19.921282 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:15:19.921648 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:19.921330 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:15:19.941223 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:19.941193 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:15:19.941431 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:19.941413 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:15:20.569670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:20.569604 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-df8cd"] Jul 29 07:15:20.569848 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:20.569796 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:20.569959 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:20.569932 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:20.574968 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:20.574933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgb4r"] Jul 29 07:15:20.575154 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:20.575089 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:20.575226 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:20.575200 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:20.582570 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:20.582532 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82z89"] Jul 29 07:15:20.582720 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:20.582676 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:20.582815 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:20.582790 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:22.731820 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:22.731769 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:22.732451 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:22.731771 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:22.732451 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:22.731896 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:22.732451 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:22.731919 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:22.732451 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:22.732020 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:22.732451 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:22.732111 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:23.931902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:23.931863 2568 generic.go:356] "Generic (PLEG): container finished" podID="087ed21b-a53d-48ce-9f53-7894005159c3" containerID="dd20e27c14fb2b5fc22d1a9306de12fc0a689306d19164ffa0377abd8ed597b0" exitCode=0 Jul 29 07:15:23.932308 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:23.931927 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerDied","Data":"dd20e27c14fb2b5fc22d1a9306de12fc0a689306d19164ffa0377abd8ed597b0"} Jul 29 07:15:24.469678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.469642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:24.469678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.469688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:24.469949 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.469713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:24.469949 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469813 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:24.469949 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469832 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:24.469949 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469854 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:24.469949 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469884 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs podName:24546c0d-8683-4aac-8dd8-2f077292bc4f nodeName:}" failed. No retries permitted until 2026-07-29 07:15:56.469863865 +0000 UTC m=+66.241314656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs") pod "network-metrics-daemon-df8cd" (UID: "24546c0d-8683-4aac-8dd8-2f077292bc4f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:24.469949 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469902 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret podName:c71bd361-1cda-4ccd-8daf-3fc786c6d773 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:56.469893229 +0000 UTC m=+66.241344007 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret") pod "global-pull-secret-syncer-82z89" (UID: "c71bd361-1cda-4ccd-8daf-3fc786c6d773") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:24.470289 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469959 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:24.470289 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.469980 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nrvhh for pod openshift-network-diagnostics/network-check-target-tgb4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:24.470289 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.470029 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh podName:4221d142-a9c3-4f22-8a63-216d86b24462 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:56.470016279 +0000 UTC m=+66.241467072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrvhh" (UniqueName: "kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh") pod "network-check-target-tgb4r" (UID: "4221d142-a9c3-4f22-8a63-216d86b24462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:24.732329 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.732230 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:24.732493 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.732352 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z89" podUID="c71bd361-1cda-4ccd-8daf-3fc786c6d773" Jul 29 07:15:24.732800 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.732776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:24.732885 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.732865 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df8cd" podUID="24546c0d-8683-4aac-8dd8-2f077292bc4f" Jul 29 07:15:24.732942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.732927 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:24.732991 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:24.732981 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgb4r" podUID="4221d142-a9c3-4f22-8a63-216d86b24462" Jul 29 07:15:24.935577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.935544 2568 generic.go:356] "Generic (PLEG): container finished" podID="087ed21b-a53d-48ce-9f53-7894005159c3" containerID="51d50cbd313e3171d882719f95682ece699c1a3f40559970dd1ebc3f9583c7f1" exitCode=0 Jul 29 07:15:24.936071 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:24.935592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerDied","Data":"51d50cbd313e3171d882719f95682ece699c1a3f40559970dd1ebc3f9583c7f1"} Jul 29 07:15:25.940691 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:25.940654 2568 generic.go:356] "Generic (PLEG): container finished" podID="087ed21b-a53d-48ce-9f53-7894005159c3" containerID="934e7bdfc426616b50d5a27fb2f22cc68c89552ceb999d65a8359bb8a8e4a206" exitCode=0 Jul 29 07:15:25.941093 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:25.940727 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerDied","Data":"934e7bdfc426616b50d5a27fb2f22cc68c89552ceb999d65a8359bb8a8e4a206"} Jul 29 07:15:26.049960 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.049929 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-73.ec2.internal" event="NodeReady" Jul 29 07:15:26.050134 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.050081 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 07:15:26.085668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.085634 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq"] Jul 29 07:15:26.088374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.088344 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" Jul 29 07:15:26.089294 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.089272 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-684dc586b6-4cddw"] Jul 29 07:15:26.091308 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.091284 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2mvnt\"" Jul 29 07:15:26.091420 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.091305 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.091420 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.091342 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.091873 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.091857 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.095334 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.095310 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 07:15:26.095334 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.095312 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-846cg\"" Jul 29 07:15:26.096499 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.096482 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 07:15:26.096685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.096661 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 07:15:26.100718 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.100676 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq"] Jul 29 07:15:26.101928 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.101807 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 07:15:26.102365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.102339 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ttnwj"] Jul 29 07:15:26.106456 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.106434 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp"] Jul 29 07:15:26.106574 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.106515 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.109508 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.109328 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-gwn5p"] Jul 29 07:15:26.109656 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.109436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.110482 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.110405 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.110482 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.110449 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 07:15:26.110688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.110504 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qjs42\"" Jul 29 07:15:26.110688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.110569 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.113395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.111493 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 07:15:26.113395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.112209 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 29 07:15:26.113395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.112440 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 29 07:15:26.113395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.112948 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lq6dv\"" Jul 29 07:15:26.113395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.113193 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.115403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.115379 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.118508 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.118480 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k"] Jul 29 07:15:26.119953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.119930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.120793 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.120765 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 07:15:26.121927 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.121907 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb"] Jul 29 07:15:26.122035 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.122020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.122877 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.122856 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 29 07:15:26.122999 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.122981 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 29 07:15:26.123295 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.123087 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2fcxq\"" Jul 29 07:15:26.123295 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.123291 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.123429 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.123373 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.124754 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.124732 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 07:15:26.124754 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.124733 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92"] Jul 29 07:15:26.124889 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.124826 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.125439 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.125421 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 07:15:26.125543 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.125471 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.125543 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.125530 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.126341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.126323 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hmwc8\"" Jul 29 07:15:26.129386 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.129359 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6549448c57-9mrc9"] Jul 29 07:15:26.129505 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.129483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.130558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.130534 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 29 07:15:26.130671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.130536 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8vmfp\"" Jul 29 07:15:26.130671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.130608 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 07:15:26.131321 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.131272 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.131582 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.131564 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.132119 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132096 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.132450 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132431 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 07:15:26.132892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132872 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-684dc586b6-4cddw"] Jul 29 07:15:26.132964 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ttnwj"] Jul 29 07:15:26.132964 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132935 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp"] Jul 29 07:15:26.132964 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132954 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k"] Jul 29 07:15:26.133084 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb"] Jul 29 07:15:26.133084 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.132990 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xkkwv"] Jul 29 07:15:26.133496 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.133459 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.135304 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.135281 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 07:15:26.135304 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.135298 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rv5xs\"" Jul 29 07:15:26.135554 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.135531 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.137423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.137401 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99"] Jul 29 07:15:26.137564 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.137548 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.138854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.138835 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r8m8t\"" Jul 29 07:15:26.139127 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.138941 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 07:15:26.139127 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.138971 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.139127 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.139082 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 07:15:26.139479 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.139462 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.139585 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.139493 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 07:15:26.139879 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.139755 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 07:15:26.140360 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.140338 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qrcg8"] Jul 29 07:15:26.140586 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.140572 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" Jul 29 07:15:26.140661 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.140599 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 07:15:26.140878 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.140860 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6q689\"" Jul 29 07:15:26.140957 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.140944 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 07:15:26.147089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.143406 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr"] Jul 29 07:15:26.147089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.143498 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.147089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.145833 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.147089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.146109 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.147089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.146324 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5g2kz\"" Jul 29 07:15:26.147089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.146746 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 07:15:26.147796 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.147598 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6scvq\"" Jul 29 07:15:26.148056 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148026 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 07:15:26.148138 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148114 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 07:15:26.148198 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148156 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92"] Jul 29 07:15:26.148249 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-gwn5p"] Jul 29 07:15:26.148249 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6549448c57-9mrc9"] Jul 29 07:15:26.148330 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148256 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qrcg8"] Jul 29 07:15:26.149449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.148369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.151102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.151079 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr"] Jul 29 07:15:26.156490 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.155962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99"] Jul 29 07:15:26.156911 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.156888 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 07:15:26.157013 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.156964 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 07:15:26.157013 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.156972 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7wclq\"" Jul 29 07:15:26.157997 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.157979 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xkkwv"] Jul 29 07:15:26.181376 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-installation-pull-secrets\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-bound-sa-token\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb2b\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-kube-api-access-vlb2b\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181470 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-trusted-ca\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-image-registry-private-configuration\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh79\" (UniqueName: \"kubernetes.io/projected/5f8893ab-e2aa-424e-9f9d-a23051175228-kube-api-access-kbh79\") pod \"volume-data-source-validator-549c6d54cf-8qjrq\" (UID: \"5f8893ab-e2aa-424e-9f9d-a23051175228\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" Jul 29 07:15:26.181667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-ca-trust-extracted\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.181667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.181627 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-certificates\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.282006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.281964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfn6\" (UniqueName: \"kubernetes.io/projected/9a5af4a0-b41c-4e64-b58e-5cafe5c66647-kube-api-access-pwfn6\") pod \"network-check-source-7d88cb55d8-xxk99\" (UID: \"9a5af4a0-b41c-4e64-b58e-5cafe5c66647\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" Jul 29 07:15:26.282006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ac2baf2-1488-4dc6-baec-f6f313f330e0-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2abf6566-aa10-4aec-bf74-26f935f88cda-serving-cert\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282064 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282095 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvg5f\" (UniqueName: \"kubernetes.io/projected/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-kube-api-access-tvg5f\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-config\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7154b5ee-985a-4b27-96e7-34cd469b9b69-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-trusted-ca\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2abf6566-aa10-4aec-bf74-26f935f88cda-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.282307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb2b\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-kube-api-access-vlb2b\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eac22f6-6f8c-499c-ac0f-efb9459915e6-tmp-dir\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-image-registry-private-configuration\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lkb\" (UniqueName: \"kubernetes.io/projected/a2ee561b-df1e-47b7-a495-c6fee6888ec5-kube-api-access-d5lkb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282420 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ee561b-df1e-47b7-a495-c6fee6888ec5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2abf6566-aa10-4aec-bf74-26f935f88cda-service-ca-bundle\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbcm\" (UniqueName: \"kubernetes.io/projected/602f7158-c486-4ca4-8be3-ca366799fe76-kube-api-access-7sbcm\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.282650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-serving-cert\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8gc\" (UniqueName: \"kubernetes.io/projected/5edede8b-3ffc-4a62-873a-8efa06389841-kube-api-access-kk8gc\") pod \"ingress-canary-qrcg8\" (UID: \"5edede8b-3ffc-4a62-873a-8efa06389841\") " pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jch\" (UniqueName: \"kubernetes.io/projected/2eac22f6-6f8c-499c-ac0f-efb9459915e6-kube-api-access-f4jch\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282726 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/602f7158-c486-4ca4-8be3-ca366799fe76-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2eac22f6-6f8c-499c-ac0f-efb9459915e6-config-volume\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2abf6566-aa10-4aec-bf74-26f935f88cda-tmp\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282898 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh5f\" (UniqueName: \"kubernetes.io/projected/2abf6566-aa10-4aec-bf74-26f935f88cda-kube-api-access-lnh5f\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-trusted-ca\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.282990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.282979 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9clh\" (UniqueName: \"kubernetes.io/projected/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-kube-api-access-m9clh\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283012 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-trusted-ca\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-installation-pull-secrets\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-bound-sa-token\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ee561b-df1e-47b7-a495-c6fee6888ec5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.283187 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.283201 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684dc586b6-4cddw: secret "image-registry-tls" not found Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-default-certificate\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.283263 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls podName:3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.783243669 +0000 UTC m=+36.554694445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls") pod "image-registry-684dc586b6-4cddw" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78") : secret "image-registry-tls" not found Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-stats-auth\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlkh\" (UniqueName: \"kubernetes.io/projected/b6e776c2-47be-45d2-9a2e-f51c83917c11-kube-api-access-lxlkh\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283340 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edede8b-3ffc-4a62-873a-8efa06389841-cert\") pod \"ingress-canary-qrcg8\" (UID: \"5edede8b-3ffc-4a62-873a-8efa06389841\") " pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2abf6566-aa10-4aec-bf74-26f935f88cda-snapshots\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh79\" (UniqueName: \"kubernetes.io/projected/5f8893ab-e2aa-424e-9f9d-a23051175228-kube-api-access-kbh79\") pod \"volume-data-source-validator-549c6d54cf-8qjrq\" (UID: \"5f8893ab-e2aa-424e-9f9d-a23051175228\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" Jul 29 07:15:26.283454 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-ca-trust-extracted\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-certificates\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksxd\" (UniqueName: \"kubernetes.io/projected/1ac2baf2-1488-4dc6-baec-f6f313f330e0-kube-api-access-rksxd\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.283922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/602f7158-c486-4ca4-8be3-ca366799fe76-config\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.283922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283695 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-ca-trust-extracted\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.283922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.283897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-certificates\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.287237 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.287205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-image-registry-private-configuration\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.287365 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.287272 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-installation-pull-secrets\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.291738 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.291530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb2b\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-kube-api-access-vlb2b\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.291920 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.291770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-bound-sa-token\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.291920 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.291787 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh79\" (UniqueName: \"kubernetes.io/projected/5f8893ab-e2aa-424e-9f9d-a23051175228-kube-api-access-kbh79\") pod \"volume-data-source-validator-549c6d54cf-8qjrq\" (UID: \"5f8893ab-e2aa-424e-9f9d-a23051175228\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" Jul 29 07:15:26.383961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.383867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2abf6566-aa10-4aec-bf74-26f935f88cda-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.383961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.383931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eac22f6-6f8c-499c-ac0f-efb9459915e6-tmp-dir\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.383975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lkb\" (UniqueName: \"kubernetes.io/projected/a2ee561b-df1e-47b7-a495-c6fee6888ec5-kube-api-access-d5lkb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ee561b-df1e-47b7-a495-c6fee6888ec5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2abf6566-aa10-4aec-bf74-26f935f88cda-service-ca-bundle\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbcm\" (UniqueName: \"kubernetes.io/projected/602f7158-c486-4ca4-8be3-ca366799fe76-kube-api-access-7sbcm\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-serving-cert\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8gc\" (UniqueName: \"kubernetes.io/projected/5edede8b-3ffc-4a62-873a-8efa06389841-kube-api-access-kk8gc\") pod \"ingress-canary-qrcg8\" (UID: \"5edede8b-3ffc-4a62-873a-8efa06389841\") " pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jch\" (UniqueName: \"kubernetes.io/projected/2eac22f6-6f8c-499c-ac0f-efb9459915e6-kube-api-access-f4jch\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.384213 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.384207 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384220 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/602f7158-c486-4ca4-8be3-ca366799fe76-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384274 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384298 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2eac22f6-6f8c-499c-ac0f-efb9459915e6-config-volume\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2abf6566-aa10-4aec-bf74-26f935f88cda-tmp\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh5f\" (UniqueName: \"kubernetes.io/projected/2abf6566-aa10-4aec-bf74-26f935f88cda-kube-api-access-lnh5f\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-trusted-ca\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384426 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eac22f6-6f8c-499c-ac0f-efb9459915e6-tmp-dir\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9clh\" (UniqueName: \"kubernetes.io/projected/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-kube-api-access-m9clh\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384481 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384519 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ee561b-df1e-47b7-a495-c6fee6888ec5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.384546 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.884527184 +0000 UTC m=+36.655977977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : configmap references non-existent config key: service-ca.crt Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-default-certificate\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.384620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-stats-auth\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlkh\" (UniqueName: \"kubernetes.io/projected/b6e776c2-47be-45d2-9a2e-f51c83917c11-kube-api-access-lxlkh\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edede8b-3ffc-4a62-873a-8efa06389841-cert\") pod \"ingress-canary-qrcg8\" (UID: \"5edede8b-3ffc-4a62-873a-8efa06389841\") " pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2abf6566-aa10-4aec-bf74-26f935f88cda-snapshots\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksxd\" (UniqueName: \"kubernetes.io/projected/1ac2baf2-1488-4dc6-baec-f6f313f330e0-kube-api-access-rksxd\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/602f7158-c486-4ca4-8be3-ca366799fe76-config\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfn6\" (UniqueName: \"kubernetes.io/projected/9a5af4a0-b41c-4e64-b58e-5cafe5c66647-kube-api-access-pwfn6\") pod \"network-check-source-7d88cb55d8-xxk99\" (UID: \"9a5af4a0-b41c-4e64-b58e-5cafe5c66647\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ac2baf2-1488-4dc6-baec-f6f313f330e0-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2abf6566-aa10-4aec-bf74-26f935f88cda-serving-cert\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384846 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2abf6566-aa10-4aec-bf74-26f935f88cda-service-ca-bundle\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384861 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384878 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ee561b-df1e-47b7-a495-c6fee6888ec5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384878 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2eac22f6-6f8c-499c-ac0f-efb9459915e6-config-volume\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.384906 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls podName:2eac22f6-6f8c-499c-ac0f-efb9459915e6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.884890368 +0000 UTC m=+36.656341144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls") pod "dns-default-xkkwv" (UID: "2eac22f6-6f8c-499c-ac0f-efb9459915e6") : secret "dns-default-metrics-tls" not found Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvg5f\" (UniqueName: \"kubernetes.io/projected/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-kube-api-access-tvg5f\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.384938 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 07:15:26.385443 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.384973 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.384997 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.884981572 +0000 UTC m=+36.656432364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : secret "router-metrics-certs-default" not found Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.384971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-config\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.385028 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls podName:1ac2baf2-1488-4dc6-baec-f6f313f330e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.885015389 +0000 UTC m=+36.656466193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lqljb" (UID: "1ac2baf2-1488-4dc6-baec-f6f313f330e0") : secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.385076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7154b5ee-985a-4b27-96e7-34cd469b9b69-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.385110 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2abf6566-aa10-4aec-bf74-26f935f88cda-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.385770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-config\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.385821 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7154b5ee-985a-4b27-96e7-34cd469b9b69-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.385831 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.385887 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert podName:7154b5ee-985a-4b27-96e7-34cd469b9b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.885872379 +0000 UTC m=+36.657323156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sqbgr" (UID: "7154b5ee-985a-4b27-96e7-34cd469b9b69") : secret "networking-console-plugin-cert" not found Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.385957 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.385969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-trusted-ca\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.386204 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.385989 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls podName:b6e776c2-47be-45d2-9a2e-f51c83917c11 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:26.885978394 +0000 UTC m=+36.657429185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls") pod "cluster-samples-operator-6748489656-5wk92" (UID: "b6e776c2-47be-45d2-9a2e-f51c83917c11") : secret "samples-operator-tls" not found Jul 29 07:15:26.386780 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.386316 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2abf6566-aa10-4aec-bf74-26f935f88cda-tmp\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.387126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.387103 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2abf6566-aa10-4aec-bf74-26f935f88cda-snapshots\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.387264 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.387157 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/602f7158-c486-4ca4-8be3-ca366799fe76-config\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.387400 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.387381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ac2baf2-1488-4dc6-baec-f6f313f330e0-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.388014 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.387971 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/602f7158-c486-4ca4-8be3-ca366799fe76-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.388389 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.388347 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ee561b-df1e-47b7-a495-c6fee6888ec5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.389113 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.389079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-default-certificate\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.389443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.389419 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-stats-auth\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.389520 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.389453 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edede8b-3ffc-4a62-873a-8efa06389841-cert\") pod \"ingress-canary-qrcg8\" (UID: \"5edede8b-3ffc-4a62-873a-8efa06389841\") " pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.390250 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.390228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-serving-cert\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.390343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.390231 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2abf6566-aa10-4aec-bf74-26f935f88cda-serving-cert\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.393862 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.393699 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lkb\" (UniqueName: \"kubernetes.io/projected/a2ee561b-df1e-47b7-a495-c6fee6888ec5-kube-api-access-d5lkb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp\" (UID: \"a2ee561b-df1e-47b7-a495-c6fee6888ec5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.395502 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.395472 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh5f\" (UniqueName: \"kubernetes.io/projected/2abf6566-aa10-4aec-bf74-26f935f88cda-kube-api-access-lnh5f\") pod \"insights-operator-5885fdb759-gwn5p\" (UID: \"2abf6566-aa10-4aec-bf74-26f935f88cda\") " pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.395668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.395649 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvg5f\" (UniqueName: \"kubernetes.io/projected/7a9c8dc2-3c50-48e2-a097-7313bfbd4de3-kube-api-access-tvg5f\") pod \"console-operator-584dcb6b54-ttnwj\" (UID: \"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3\") " pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.397775 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.397719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9clh\" (UniqueName: \"kubernetes.io/projected/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-kube-api-access-m9clh\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.399746 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.399692 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jch\" (UniqueName: \"kubernetes.io/projected/2eac22f6-6f8c-499c-ac0f-efb9459915e6-kube-api-access-f4jch\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.400098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.399890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" Jul 29 07:15:26.400338 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.400313 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8gc\" (UniqueName: \"kubernetes.io/projected/5edede8b-3ffc-4a62-873a-8efa06389841-kube-api-access-kk8gc\") pod \"ingress-canary-qrcg8\" (UID: \"5edede8b-3ffc-4a62-873a-8efa06389841\") " pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.400608 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.400587 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbcm\" (UniqueName: \"kubernetes.io/projected/602f7158-c486-4ca4-8be3-ca366799fe76-kube-api-access-7sbcm\") pod \"service-ca-operator-5d6c75f84f-jbm7k\" (UID: \"602f7158-c486-4ca4-8be3-ca366799fe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.400767 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.400746 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfn6\" (UniqueName: \"kubernetes.io/projected/9a5af4a0-b41c-4e64-b58e-5cafe5c66647-kube-api-access-pwfn6\") pod \"network-check-source-7d88cb55d8-xxk99\" (UID: \"9a5af4a0-b41c-4e64-b58e-5cafe5c66647\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" Jul 29 07:15:26.400854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.400833 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksxd\" (UniqueName: \"kubernetes.io/projected/1ac2baf2-1488-4dc6-baec-f6f313f330e0-kube-api-access-rksxd\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.400901 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.400860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlkh\" (UniqueName: \"kubernetes.io/projected/b6e776c2-47be-45d2-9a2e-f51c83917c11-kube-api-access-lxlkh\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.424680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.424642 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:26.436010 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.435983 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" Jul 29 07:15:26.446493 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.446460 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" Jul 29 07:15:26.461628 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.461255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" Jul 29 07:15:26.491433 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.491151 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" Jul 29 07:15:26.496745 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.496658 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qrcg8" Jul 29 07:15:26.679472 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.679205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq"] Jul 29 07:15:26.716293 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.716258 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ttnwj"] Jul 29 07:15:26.722294 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:26.722256 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9c8dc2_3c50_48e2_a097_7313bfbd4de3.slice/crio-58266942e1c5c36852881fecce58f85b01ee65bad2d5340226c0b6917f4ba19a WatchSource:0}: Error finding container 58266942e1c5c36852881fecce58f85b01ee65bad2d5340226c0b6917f4ba19a: Status 404 returned error can't find the container with id 58266942e1c5c36852881fecce58f85b01ee65bad2d5340226c0b6917f4ba19a Jul 29 07:15:26.732229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.732197 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:26.732716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.732196 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:26.733102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.732197 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:26.737193 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.737166 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 07:15:26.737652 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.737629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-gwn5p"] Jul 29 07:15:26.737756 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.737669 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rkrh4\"" Jul 29 07:15:26.737809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.737774 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb4zn\"" Jul 29 07:15:26.738839 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.738795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp"] Jul 29 07:15:26.741631 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.741603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 07:15:26.745366 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:26.745334 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf6566_aa10_4aec_bf74_26f935f88cda.slice/crio-323f4de24f27ead3c99715cf24654f4a9017e0f7aa82df161daa2283d6c5fe8b WatchSource:0}: Error finding container 323f4de24f27ead3c99715cf24654f4a9017e0f7aa82df161daa2283d6c5fe8b: Status 404 returned error can't find the container with id 323f4de24f27ead3c99715cf24654f4a9017e0f7aa82df161daa2283d6c5fe8b Jul 29 07:15:26.747142 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:26.747109 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ee561b_df1e_47b7_a495_c6fee6888ec5.slice/crio-794c5ed61a113fea69b5028944155b4d362b00eb512231b092c5996aefe0fd93 WatchSource:0}: Error finding container 794c5ed61a113fea69b5028944155b4d362b00eb512231b092c5996aefe0fd93: Status 404 returned error can't find the container with id 794c5ed61a113fea69b5028944155b4d362b00eb512231b092c5996aefe0fd93 Jul 29 07:15:26.788655 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.788612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:26.788823 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.788780 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 07:15:26.788823 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.788798 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684dc586b6-4cddw: secret "image-registry-tls" not found Jul 29 07:15:26.788928 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.788871 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls podName:3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.788847857 +0000 UTC m=+37.560298638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls") pod "image-registry-684dc586b6-4cddw" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78") : secret "image-registry-tls" not found Jul 29 07:15:26.803444 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.803413 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qrcg8"] Jul 29 07:15:26.804878 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.804795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99"] Jul 29 07:15:26.805985 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:26.805958 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edede8b_3ffc_4a62_873a_8efa06389841.slice/crio-f71013b846f296f6af9357f7bb0d99a6064846c98307b6ca903ccb02d18c2704 WatchSource:0}: Error finding container f71013b846f296f6af9357f7bb0d99a6064846c98307b6ca903ccb02d18c2704: Status 404 returned error can't find the container with id f71013b846f296f6af9357f7bb0d99a6064846c98307b6ca903ccb02d18c2704 Jul 29 07:15:26.806996 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:26.806950 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5af4a0_b41c_4e64_b58e_5cafe5c66647.slice/crio-a342fc83d0e7e97a7f94de31a720b8a820a7a0d587a482d72b54f78b4316f8bc WatchSource:0}: Error finding container a342fc83d0e7e97a7f94de31a720b8a820a7a0d587a482d72b54f78b4316f8bc: Status 404 returned error can't find the container with id a342fc83d0e7e97a7f94de31a720b8a820a7a0d587a482d72b54f78b4316f8bc Jul 29 07:15:26.807417 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.806029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k"] Jul 29 07:15:26.810142 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:26.810034 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602f7158_c486_4ca4_8be3_ca366799fe76.slice/crio-be4c7d6e4e0b3bd3a6979f03bc9f510e46dfdcec33a23349e4f0ff3426a150ec WatchSource:0}: Error finding container be4c7d6e4e0b3bd3a6979f03bc9f510e46dfdcec33a23349e4f0ff3426a150ec: Status 404 returned error can't find the container with id be4c7d6e4e0b3bd3a6979f03bc9f510e46dfdcec33a23349e4f0ff3426a150ec Jul 29 07:15:26.889377 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.889276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.889558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.889406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:26.889558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.889445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:26.889558 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889474 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 07:15:26.889558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.889488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:26.889558 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889513 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 07:15:26.889558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.889526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.889563 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889574 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889584 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls podName:2eac22f6-6f8c-499c-ac0f-efb9459915e6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.889564555 +0000 UTC m=+37.661015350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls") pod "dns-default-xkkwv" (UID: "2eac22f6-6f8c-499c-ac0f-efb9459915e6") : secret "dns-default-metrics-tls" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889639 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls podName:1ac2baf2-1488-4dc6-baec-f6f313f330e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.889623307 +0000 UTC m=+37.661074084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lqljb" (UID: "1ac2baf2-1488-4dc6-baec-f6f313f330e0") : secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889660 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.889650267 +0000 UTC m=+37.661101052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : secret "router-metrics-certs-default" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889690 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.889669176 +0000 UTC m=+37.661120053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : configmap references non-existent config key: service-ca.crt Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889732 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889748 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889765 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert podName:7154b5ee-985a-4b27-96e7-34cd469b9b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.889755688 +0000 UTC m=+37.661206493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sqbgr" (UID: "7154b5ee-985a-4b27-96e7-34cd469b9b69") : secret "networking-console-plugin-cert" not found Jul 29 07:15:26.889852 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:26.889807 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls podName:b6e776c2-47be-45d2-9a2e-f51c83917c11 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:27.889778752 +0000 UTC m=+37.661229548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls") pod "cluster-samples-operator-6748489656-5wk92" (UID: "b6e776c2-47be-45d2-9a2e-f51c83917c11") : secret "samples-operator-tls" not found Jul 29 07:15:26.944872 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.944837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" event={"ID":"602f7158-c486-4ca4-8be3-ca366799fe76","Type":"ContainerStarted","Data":"be4c7d6e4e0b3bd3a6979f03bc9f510e46dfdcec33a23349e4f0ff3426a150ec"} Jul 29 07:15:26.946319 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.946282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qrcg8" event={"ID":"5edede8b-3ffc-4a62-873a-8efa06389841","Type":"ContainerStarted","Data":"f71013b846f296f6af9357f7bb0d99a6064846c98307b6ca903ccb02d18c2704"} Jul 29 07:15:26.947609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.947576 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" event={"ID":"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3","Type":"ContainerStarted","Data":"58266942e1c5c36852881fecce58f85b01ee65bad2d5340226c0b6917f4ba19a"} Jul 29 07:15:26.948899 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.948871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" event={"ID":"2abf6566-aa10-4aec-bf74-26f935f88cda","Type":"ContainerStarted","Data":"323f4de24f27ead3c99715cf24654f4a9017e0f7aa82df161daa2283d6c5fe8b"} Jul 29 07:15:26.950376 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.950347 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" event={"ID":"5f8893ab-e2aa-424e-9f9d-a23051175228","Type":"ContainerStarted","Data":"fa739d4c982e31b481c807cc66043fb6ba2dbb809e99515eaf55359bbbd43d9d"} Jul 29 07:15:26.951579 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.951553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" event={"ID":"9a5af4a0-b41c-4e64-b58e-5cafe5c66647","Type":"ContainerStarted","Data":"a342fc83d0e7e97a7f94de31a720b8a820a7a0d587a482d72b54f78b4316f8bc"} Jul 29 07:15:26.952750 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:26.952725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" event={"ID":"a2ee561b-df1e-47b7-a495-c6fee6888ec5","Type":"ContainerStarted","Data":"794c5ed61a113fea69b5028944155b4d362b00eb512231b092c5996aefe0fd93"} Jul 29 07:15:27.799057 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.799001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:27.799239 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.799220 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 07:15:27.799239 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.799238 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684dc586b6-4cddw: secret "image-registry-tls" not found Jul 29 07:15:27.799360 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.799297 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls podName:3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.799278557 +0000 UTC m=+39.570729335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls") pod "image-registry-684dc586b6-4cddw" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78") : secret "image-registry-tls" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.900520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.900587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.900621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.900664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.900703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:27.900773 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.900920 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.900986 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.900965542 +0000 UTC m=+39.672416338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : secret "router-metrics-certs-default" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901474 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901527 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls podName:2eac22f6-6f8c-499c-ac0f-efb9459915e6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.901510743 +0000 UTC m=+39.672961538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls") pod "dns-default-xkkwv" (UID: "2eac22f6-6f8c-499c-ac0f-efb9459915e6") : secret "dns-default-metrics-tls" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901594 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901629 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls podName:1ac2baf2-1488-4dc6-baec-f6f313f330e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.901617826 +0000 UTC m=+39.673068608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lqljb" (UID: "1ac2baf2-1488-4dc6-baec-f6f313f330e0") : secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901699 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.901688662 +0000 UTC m=+39.673139439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : configmap references non-existent config key: service-ca.crt Jul 29 07:15:27.901936 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901773 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 07:15:27.902768 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901807 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert podName:7154b5ee-985a-4b27-96e7-34cd469b9b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.901796106 +0000 UTC m=+39.673246887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sqbgr" (UID: "7154b5ee-985a-4b27-96e7-34cd469b9b69") : secret "networking-console-plugin-cert" not found Jul 29 07:15:27.902768 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901866 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 07:15:27.902768 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:27.901896 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls podName:b6e776c2-47be-45d2-9a2e-f51c83917c11 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:29.901886318 +0000 UTC m=+39.673337097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls") pod "cluster-samples-operator-6748489656-5wk92" (UID: "b6e776c2-47be-45d2-9a2e-f51c83917c11") : secret "samples-operator-tls" not found Jul 29 07:15:29.825828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.825787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:29.826335 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.826117 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 07:15:29.826335 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.826139 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684dc586b6-4cddw: secret "image-registry-tls" not found Jul 29 07:15:29.826335 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.826200 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls podName:3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.826180594 +0000 UTC m=+43.597631377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls") pod "image-registry-684dc586b6-4cddw" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78") : secret "image-registry-tls" not found Jul 29 07:15:29.927208 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.927170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:29.927360 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.927273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:29.927360 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.927312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:29.927360 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.927342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:29.927480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.927384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:29.927480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:29.927421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:29.927612 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.927595 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 07:15:29.927680 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.927664 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls podName:b6e776c2-47be-45d2-9a2e-f51c83917c11 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.927644589 +0000 UTC m=+43.699095372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls") pod "cluster-samples-operator-6748489656-5wk92" (UID: "b6e776c2-47be-45d2-9a2e-f51c83917c11") : secret "samples-operator-tls" not found Jul 29 07:15:29.928081 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928063 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 07:15:29.928199 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928103 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.928092277 +0000 UTC m=+43.699543053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : secret "router-metrics-certs-default" not found Jul 29 07:15:29.928199 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928160 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 07:15:29.928199 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928196 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls podName:2eac22f6-6f8c-499c-ac0f-efb9459915e6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.928184123 +0000 UTC m=+43.699634910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls") pod "dns-default-xkkwv" (UID: "2eac22f6-6f8c-499c-ac0f-efb9459915e6") : secret "dns-default-metrics-tls" not found Jul 29 07:15:29.928328 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928247 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:29.928328 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928273 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls podName:1ac2baf2-1488-4dc6-baec-f6f313f330e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.928265111 +0000 UTC m=+43.699715887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lqljb" (UID: "1ac2baf2-1488-4dc6-baec-f6f313f330e0") : secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:29.928328 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928315 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.928309385 +0000 UTC m=+43.699760161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : configmap references non-existent config key: service-ca.crt Jul 29 07:15:29.928429 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928346 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 07:15:29.928429 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:29.928364 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert podName:7154b5ee-985a-4b27-96e7-34cd469b9b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:33.928358238 +0000 UTC m=+43.699809014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sqbgr" (UID: "7154b5ee-985a-4b27-96e7-34cd469b9b69") : secret "networking-console-plugin-cert" not found Jul 29 07:15:31.896236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:31.896183 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:31.982420 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:31.982383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:33.867356 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.867257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:33.867774 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.867413 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 07:15:33.867774 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.867438 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684dc586b6-4cddw: secret "image-registry-tls" not found Jul 29 07:15:33.867774 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.867503 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls podName:3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.867485026 +0000 UTC m=+51.638935808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls") pod "image-registry-684dc586b6-4cddw" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78") : secret "image-registry-tls" not found Jul 29 07:15:33.968399 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.968350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:33.968399 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.968405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:33.968663 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.968520 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:33.968663 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.968526 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.968506913 +0000 UTC m=+51.739957690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : configmap references non-existent config key: service-ca.crt Jul 29 07:15:33.968663 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.968632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:33.968822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.968688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:33.968822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.968790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:33.968928 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:33.968884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:33.968928 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.968917 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 07:15:33.969028 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.968959 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert podName:7154b5ee-985a-4b27-96e7-34cd469b9b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.96894374 +0000 UTC m=+51.740394518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sqbgr" (UID: "7154b5ee-985a-4b27-96e7-34cd469b9b69") : secret "networking-console-plugin-cert" not found Jul 29 07:15:33.969028 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.968986 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls podName:1ac2baf2-1488-4dc6-baec-f6f313f330e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.968973227 +0000 UTC m=+51.740424008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lqljb" (UID: "1ac2baf2-1488-4dc6-baec-f6f313f330e0") : secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:33.969028 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.968995 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 07:15:33.969284 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.969066 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 07:15:33.969284 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.969069 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls podName:2eac22f6-6f8c-499c-ac0f-efb9459915e6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.969025417 +0000 UTC m=+51.740476200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls") pod "dns-default-xkkwv" (UID: "2eac22f6-6f8c-499c-ac0f-efb9459915e6") : secret "dns-default-metrics-tls" not found Jul 29 07:15:33.969284 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.969104 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls podName:b6e776c2-47be-45d2-9a2e-f51c83917c11 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.969096824 +0000 UTC m=+51.740547601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls") pod "cluster-samples-operator-6748489656-5wk92" (UID: "b6e776c2-47be-45d2-9a2e-f51c83917c11") : secret "samples-operator-tls" not found Jul 29 07:15:33.969284 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.969112 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 07:15:33.969284 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:33.969137 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:41.969129809 +0000 UTC m=+51.740580586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : secret "router-metrics-certs-default" not found Jul 29 07:15:35.927175 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:35.927135 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:35.990527 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:35.990489 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:37.003654 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.003564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qrcg8" event={"ID":"5edede8b-3ffc-4a62-873a-8efa06389841","Type":"ContainerStarted","Data":"832f7ce8c542321f72dafb3f4c8993c1bb7b2db7dbd2ce71995e95448ad0a8d5"} Jul 29 07:15:37.006629 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.006603 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/0.log" Jul 29 07:15:37.006809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.006642 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a9c8dc2-3c50-48e2-a097-7313bfbd4de3" containerID="8063c89fc1d99ff70073ea47a02b91c8685ec72624c7205d76334c9eab121e1a" exitCode=255 Jul 29 07:15:37.006809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.006760 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" event={"ID":"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3","Type":"ContainerDied","Data":"8063c89fc1d99ff70073ea47a02b91c8685ec72624c7205d76334c9eab121e1a"} Jul 29 07:15:37.007661 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.007151 2568 scope.go:117] "RemoveContainer" containerID="8063c89fc1d99ff70073ea47a02b91c8685ec72624c7205d76334c9eab121e1a" Jul 29 07:15:37.011734 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.011658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" event={"ID":"2abf6566-aa10-4aec-bf74-26f935f88cda","Type":"ContainerStarted","Data":"0da43505d39d740a9bbabb4863aa460e79e4a110a9e26be37f721b14a2e00420"} Jul 29 07:15:37.022827 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.022555 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qrcg8" podStartSLOduration=1.090353688 podStartE2EDuration="11.022532029s" totalImagesPullingTime="9.932178341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:37.021650356 +0000 UTC m=+46.793101156" watchObservedRunningTime="2026-07-29 07:15:37.022532029 +0000 UTC m=+46.793982829" Jul 29 07:15:37.028758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.028711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" event={"ID":"5f8893ab-e2aa-424e-9f9d-a23051175228","Type":"ContainerStarted","Data":"b10d835e511bfc1e6ac2c33c1506add4e8cecfe366c9217e80d3758871b42154"} Jul 29 07:15:37.060584 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.059938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" event={"ID":"9a5af4a0-b41c-4e64-b58e-5cafe5c66647","Type":"ContainerStarted","Data":"4e60c6e627c786ab121e9a75a5b56d0fe4a3680c608f525a073ff5f238523807"} Jul 29 07:15:37.063063 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.062877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" event={"ID":"a2ee561b-df1e-47b7-a495-c6fee6888ec5","Type":"ContainerStarted","Data":"e6188278ce74df19283114846eccbb555a4f60d26174dd86c9e7e714cb52b80b"} Jul 29 07:15:37.070007 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.065020 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" podStartSLOduration=17.073059637 podStartE2EDuration="27.065001344s" totalImagesPullingTime="9.991941707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:37.064487673 +0000 UTC m=+46.835938473" watchObservedRunningTime="2026-07-29 07:15:37.065001344 +0000 UTC m=+46.836452146" Jul 29 07:15:37.088362 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.086632 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8qjrq" podStartSLOduration=17.304132786 podStartE2EDuration="27.086609269s" totalImagesPullingTime="9.782476483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:37.085440872 +0000 UTC m=+46.856891673" watchObservedRunningTime="2026-07-29 07:15:37.086609269 +0000 UTC m=+46.858060069" Jul 29 07:15:37.130096 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.129569 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xxk99" podStartSLOduration=17.198902627 podStartE2EDuration="27.129547422s" totalImagesPullingTime="9.930644795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:37.128703889 +0000 UTC m=+46.900154687" watchObservedRunningTime="2026-07-29 07:15:37.129547422 +0000 UTC m=+46.900998222" Jul 29 07:15:37.130311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.130256 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" podStartSLOduration=17.139945301 podStartE2EDuration="27.130240923s" totalImagesPullingTime="9.990295622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:37.106493254 +0000 UTC m=+46.877944053" watchObservedRunningTime="2026-07-29 07:15:37.130240923 +0000 UTC m=+46.901691722" Jul 29 07:15:37.214399 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:37.214301 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:38.071711 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.071620 2568 generic.go:356] "Generic (PLEG): container finished" podID="087ed21b-a53d-48ce-9f53-7894005159c3" containerID="9a85abb04fcbbf454da53691b69d1ad0cba6fccb5489d2151360c6e391fe63ef" exitCode=0 Jul 29 07:15:38.072209 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.071715 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerDied","Data":"9a85abb04fcbbf454da53691b69d1ad0cba6fccb5489d2151360c6e391fe63ef"} Jul 29 07:15:38.073512 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.073484 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" event={"ID":"602f7158-c486-4ca4-8be3-ca366799fe76","Type":"ContainerStarted","Data":"adac4ebb1f281617466c587b03a8971c2ab6c7626666534bdba2a6a110b9e693"} Jul 29 07:15:38.076006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.075986 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:15:38.076501 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.076484 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/0.log" Jul 29 07:15:38.076584 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.076524 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a9c8dc2-3c50-48e2-a097-7313bfbd4de3" containerID="7ee33eecb52ed2021fa3ef905a4cf9eefeada215c9c3721e5c0d24c6f0f5eb0e" exitCode=255 Jul 29 07:15:38.076638 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.076578 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" event={"ID":"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3","Type":"ContainerDied","Data":"7ee33eecb52ed2021fa3ef905a4cf9eefeada215c9c3721e5c0d24c6f0f5eb0e"} Jul 29 07:15:38.076689 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.076672 2568 scope.go:117] "RemoveContainer" containerID="8063c89fc1d99ff70073ea47a02b91c8685ec72624c7205d76334c9eab121e1a" Jul 29 07:15:38.077387 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.076886 2568 scope.go:117] "RemoveContainer" containerID="7ee33eecb52ed2021fa3ef905a4cf9eefeada215c9c3721e5c0d24c6f0f5eb0e" Jul 29 07:15:38.077387 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:38.077114 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ttnwj_openshift-console-operator(7a9c8dc2-3c50-48e2-a097-7313bfbd4de3)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" podUID="7a9c8dc2-3c50-48e2-a097-7313bfbd4de3" Jul 29 07:15:38.137150 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.137081 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" podStartSLOduration=18.206270478 podStartE2EDuration="28.137028408s" totalImagesPullingTime="9.93075793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:38.134494685 +0000 UTC m=+47.905945476" watchObservedRunningTime="2026-07-29 07:15:38.137028408 +0000 UTC m=+47.908479212" Jul 29 07:15:38.522826 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.522783 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x7fzh"] Jul 29 07:15:38.535869 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.535839 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.536625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.536539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7fzh"] Jul 29 07:15:38.538637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.538463 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zj4vm\"" Jul 29 07:15:38.538637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.538509 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 07:15:38.538637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.538527 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 07:15:38.557458 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.557416 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:38.621569 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.621530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:38.625564 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.625530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2613fe87-dc24-45d8-8897-f211058dbd51-crio-socket\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.625762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.625577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.625762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.625603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wb6\" (UniqueName: \"kubernetes.io/projected/2613fe87-dc24-45d8-8897-f211058dbd51-kube-api-access-r8wb6\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.625762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.625695 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2613fe87-dc24-45d8-8897-f211058dbd51-data-volume\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.625762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.625750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.726386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2613fe87-dc24-45d8-8897-f211058dbd51-crio-socket\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.726430 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.726452 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wb6\" (UniqueName: \"kubernetes.io/projected/2613fe87-dc24-45d8-8897-f211058dbd51-kube-api-access-r8wb6\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726684 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:38.726575 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 07:15:38.726684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.726632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2613fe87-dc24-45d8-8897-f211058dbd51-data-volume\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726684 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:38.726643 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:39.226625625 +0000 UTC m=+48.998076403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : secret "insights-runtime-extractor-tls" not found Jul 29 07:15:38.726684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.726665 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2613fe87-dc24-45d8-8897-f211058dbd51-crio-socket\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726935 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.726705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.726935 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:38.726778 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 07:15:38.726935 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:38.726854 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:39.226837588 +0000 UTC m=+48.998288404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : configmap "kube-rbac-proxy" not found Jul 29 07:15:38.727135 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.727079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2613fe87-dc24-45d8-8897-f211058dbd51-data-volume\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.738232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.738198 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wb6\" (UniqueName: \"kubernetes.io/projected/2613fe87-dc24-45d8-8897-f211058dbd51-kube-api-access-r8wb6\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:38.864347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.864223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:38.866839 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.866804 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg"] Jul 29 07:15:38.884148 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.884109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg"] Jul 29 07:15:38.884339 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.884259 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" Jul 29 07:15:38.887094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.887067 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:38.887261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.887097 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 29 07:15:38.887261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.887124 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lfk46\"" Jul 29 07:15:38.928926 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.928884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfldr\" (UniqueName: \"kubernetes.io/projected/e91e5b20-9602-4675-8adb-0665c161de9a-kube-api-access-zfldr\") pod \"migrator-6865c9967-69wtg\" (UID: \"e91e5b20-9602-4675-8adb-0665c161de9a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" Jul 29 07:15:38.959357 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:38.959320 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:39.030225 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.030174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfldr\" (UniqueName: \"kubernetes.io/projected/e91e5b20-9602-4675-8adb-0665c161de9a-kube-api-access-zfldr\") pod \"migrator-6865c9967-69wtg\" (UID: \"e91e5b20-9602-4675-8adb-0665c161de9a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" Jul 29 07:15:39.042304 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.042258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfldr\" (UniqueName: \"kubernetes.io/projected/e91e5b20-9602-4675-8adb-0665c161de9a-kube-api-access-zfldr\") pod \"migrator-6865c9967-69wtg\" (UID: \"e91e5b20-9602-4675-8adb-0665c161de9a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" Jul 29 07:15:39.059618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.059580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:39.082684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.082656 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:15:39.083122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.083073 2568 scope.go:117] "RemoveContainer" containerID="7ee33eecb52ed2021fa3ef905a4cf9eefeada215c9c3721e5c0d24c6f0f5eb0e" Jul 29 07:15:39.083317 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:39.083289 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ttnwj_openshift-console-operator(7a9c8dc2-3c50-48e2-a097-7313bfbd4de3)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" podUID="7a9c8dc2-3c50-48e2-a097-7313bfbd4de3" Jul 29 07:15:39.086029 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.086000 2568 generic.go:356] "Generic (PLEG): container finished" podID="087ed21b-a53d-48ce-9f53-7894005159c3" containerID="13fa9d079f29eaa79ad260669afe98324af7265280759138fc409e354c72e44a" exitCode=0 Jul 29 07:15:39.086213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.086074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerDied","Data":"13fa9d079f29eaa79ad260669afe98324af7265280759138fc409e354c72e44a"} Jul 29 07:15:39.126625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.126583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:39.195569 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.195522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" Jul 29 07:15:39.232651 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.232610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:39.232841 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.232714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:39.232841 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:39.232765 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 07:15:39.232922 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:39.232838 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 07:15:39.232922 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:39.232853 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:40.232829653 +0000 UTC m=+50.004280430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : configmap "kube-rbac-proxy" not found Jul 29 07:15:39.232922 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:39.232889 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:40.2328736 +0000 UTC m=+50.004324397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : secret "insights-runtime-extractor-tls" not found Jul 29 07:15:39.326460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.326407 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:39.344369 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.344332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg"] Jul 29 07:15:39.347225 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:39.347189 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91e5b20_9602_4675_8adb_0665c161de9a.slice/crio-b3009bb30c6834d979c45da9e59b45d4029ff33893a3744a18d6d7ad23e3abdd WatchSource:0}: Error finding container b3009bb30c6834d979c45da9e59b45d4029ff33893a3744a18d6d7ad23e3abdd: Status 404 returned error can't find the container with id b3009bb30c6834d979c45da9e59b45d4029ff33893a3744a18d6d7ad23e3abdd Jul 29 07:15:39.429062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.429010 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:39.918696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:39.918668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z22p2_35d5b11f-9aca-4a3d-a010-c9cde5be22d4/dns-node-resolver/0.log" Jul 29 07:15:40.094782 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.094745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj742" event={"ID":"087ed21b-a53d-48ce-9f53-7894005159c3","Type":"ContainerStarted","Data":"3ecdceb131068c8d3929306ec1732c53f62ac5868f87e3723f1c66ed518555dc"} Jul 29 07:15:40.096283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.096249 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" event={"ID":"e91e5b20-9602-4675-8adb-0665c161de9a","Type":"ContainerStarted","Data":"b3009bb30c6834d979c45da9e59b45d4029ff33893a3744a18d6d7ad23e3abdd"} Jul 29 07:15:40.119428 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.119360 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mj742" podStartSLOduration=7.953190001 podStartE2EDuration="49.119333062s" totalImagesPullingTime="41.166143061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:40.1166518 +0000 UTC m=+49.888102600" watchObservedRunningTime="2026-07-29 07:15:40.119333062 +0000 UTC m=+49.890783863" Jul 29 07:15:40.237133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.236995 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:40.241581 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.241540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:40.241769 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:40.241662 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 07:15:40.241769 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.241720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:40.241769 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:40.241733 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:42.241713678 +0000 UTC m=+52.013164455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : configmap "kube-rbac-proxy" not found Jul 29 07:15:40.241939 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:40.241818 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 07:15:40.241939 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:40.241868 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:42.241854211 +0000 UTC m=+52.013304993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : secret "insights-runtime-extractor-tls" not found Jul 29 07:15:40.337226 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.337187 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:40.720872 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.720844 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8zxgt_9045afd8-91a2-4b61-8424-19c537bd41ee/node-ca/0.log" Jul 29 07:15:40.916239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.916200 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4sgv6"] Jul 29 07:15:40.939949 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.939916 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:40.942104 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.942068 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4sgv6"] Jul 29 07:15:40.944264 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.944230 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-psmmz\"" Jul 29 07:15:40.944401 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.944236 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 29 07:15:40.944673 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.944528 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 29 07:15:40.944673 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.944664 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 29 07:15:40.945067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.944849 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 29 07:15:40.994672 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:40.994568 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.050979 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.050937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58a839ef-90ed-4989-939b-69b301a2ae4c-signing-cabundle\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.051193 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.051025 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszxl\" (UniqueName: \"kubernetes.io/projected/58a839ef-90ed-4989-939b-69b301a2ae4c-kube-api-access-lszxl\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.051193 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.051181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58a839ef-90ed-4989-939b-69b301a2ae4c-signing-key\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.068720 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.068588 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.148716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.148670 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.152394 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.152362 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58a839ef-90ed-4989-939b-69b301a2ae4c-signing-key\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.152553 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.152509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58a839ef-90ed-4989-939b-69b301a2ae4c-signing-cabundle\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.152603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.152569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszxl\" (UniqueName: \"kubernetes.io/projected/58a839ef-90ed-4989-939b-69b301a2ae4c-kube-api-access-lszxl\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.153264 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.153185 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58a839ef-90ed-4989-939b-69b301a2ae4c-signing-cabundle\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.155097 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.155072 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58a839ef-90ed-4989-939b-69b301a2ae4c-signing-key\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.166090 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.166058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszxl\" (UniqueName: \"kubernetes.io/projected/58a839ef-90ed-4989-939b-69b301a2ae4c-kube-api-access-lszxl\") pod \"service-ca-658d7b7c78-4sgv6\" (UID: \"58a839ef-90ed-4989-939b-69b301a2ae4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.220613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.219763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.251921 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.251900 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" Jul 29 07:15:41.318436 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.318395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.321926 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.320747 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qrcg8_5edede8b-3ffc-4a62-873a-8efa06389841/serve-healthcheck-canary/0.log" Jul 29 07:15:41.399876 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.399841 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4sgv6"] Jul 29 07:15:41.403438 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:41.403408 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a839ef_90ed_4989_939b_69b301a2ae4c.slice/crio-12714504c784ddf55f3a9df4a2a8b68d38b2b753f60ae78deee68fd400fd4a35 WatchSource:0}: Error finding container 12714504c784ddf55f3a9df4a2a8b68d38b2b753f60ae78deee68fd400fd4a35: Status 404 returned error can't find the container with id 12714504c784ddf55f3a9df4a2a8b68d38b2b753f60ae78deee68fd400fd4a35 Jul 29 07:15:41.406670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.406642 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.485459 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.485428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.795274 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.795173 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.898272 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.898241 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:41.961625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.961587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:41.961773 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:41.961753 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 07:15:41.961773 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:41.961769 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684dc586b6-4cddw: secret "image-registry-tls" not found Jul 29 07:15:41.961865 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:41.961828 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls podName:3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:57.961810537 +0000 UTC m=+67.733261331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls") pod "image-registry-684dc586b6-4cddw" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78") : secret "image-registry-tls" not found Jul 29 07:15:41.969671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:41.969630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:42.062294 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.062199 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:42.062443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.062302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:42.062443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.062362 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:42.062443 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062371 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 07:15:42.062443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.062395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:42.062443 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062431 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:58.062414108 +0000 UTC m=+67.833864884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : secret "router-metrics-certs-default" not found Jul 29 07:15:42.062443 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062438 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.062468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062488 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls podName:2eac22f6-6f8c-499c-ac0f-efb9459915e6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:58.062471418 +0000 UTC m=+67.833922195 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls") pod "dns-default-xkkwv" (UID: "2eac22f6-6f8c-499c-ac0f-efb9459915e6") : secret "dns-default-metrics-tls" not found Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062512 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle podName:1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:58.062502656 +0000 UTC m=+67.833953436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle") pod "router-default-6549448c57-9mrc9" (UID: "1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7") : configmap references non-existent config key: service-ca.crt Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062534 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.062545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062546 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062570 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert podName:7154b5ee-985a-4b27-96e7-34cd469b9b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:58.062557463 +0000 UTC m=+67.834008240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sqbgr" (UID: "7154b5ee-985a-4b27-96e7-34cd469b9b69") : secret "networking-console-plugin-cert" not found Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062582 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls podName:1ac2baf2-1488-4dc6-baec-f6f313f330e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:58.062576172 +0000 UTC m=+67.834026949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lqljb" (UID: "1ac2baf2-1488-4dc6-baec-f6f313f330e0") : secret "cluster-monitoring-operator-tls" not found Jul 29 07:15:42.062728 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062604 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 07:15:42.063007 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.062747 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls podName:b6e776c2-47be-45d2-9a2e-f51c83917c11 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:58.062716622 +0000 UTC m=+67.834167399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls") pod "cluster-samples-operator-6748489656-5wk92" (UID: "b6e776c2-47be-45d2-9a2e-f51c83917c11") : secret "samples-operator-tls" not found Jul 29 07:15:42.107103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.107058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" event={"ID":"e91e5b20-9602-4675-8adb-0665c161de9a","Type":"ContainerStarted","Data":"5baa3ca36333aaa3c90ed53014ca34b2cc36ae5568efe4ac8e96dc1fa8376c6e"} Jul 29 07:15:42.107304 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.107128 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" event={"ID":"e91e5b20-9602-4675-8adb-0665c161de9a","Type":"ContainerStarted","Data":"ea1c76f5a4813eef67fd9ff2ec5237668c90b2ed61c55b70e6e27613e6284c2c"} Jul 29 07:15:42.108604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.108578 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" event={"ID":"58a839ef-90ed-4989-939b-69b301a2ae4c","Type":"ContainerStarted","Data":"0024abb4e92faaf4552a8bff4fa7ffb431d57aa63b8b3260df1e582ace77ad3b"} Jul 29 07:15:42.108723 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.108625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" event={"ID":"58a839ef-90ed-4989-939b-69b301a2ae4c","Type":"ContainerStarted","Data":"12714504c784ddf55f3a9df4a2a8b68d38b2b753f60ae78deee68fd400fd4a35"} Jul 29 07:15:42.135054 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.134979 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-69wtg" podStartSLOduration=2.448307383 podStartE2EDuration="4.134959884s" totalImagesPullingTime="1.686652501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:42.134319612 +0000 UTC m=+51.905770417" watchObservedRunningTime="2026-07-29 07:15:42.134959884 +0000 UTC m=+51.906410684" Jul 29 07:15:42.154857 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.154801 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-4sgv6" podStartSLOduration=2.154788116 podStartE2EDuration="2.154788116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:42.153542159 +0000 UTC m=+51.924992960" watchObservedRunningTime="2026-07-29 07:15:42.154788116 +0000 UTC m=+51.926238915" Jul 29 07:15:42.264954 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.264919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:42.265164 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.265005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:42.265164 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.265110 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 07:15:42.265164 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.265112 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 07:15:42.265164 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.265165 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:46.265152603 +0000 UTC m=+56.036603380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : configmap "kube-rbac-proxy" not found Jul 29 07:15:42.265357 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:42.265188 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:46.265172365 +0000 UTC m=+56.036623142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : secret "insights-runtime-extractor-tls" not found Jul 29 07:15:42.553652 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.553608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:42.946354 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:42.946319 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:43.050378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:43.050340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:43.159865 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:43.159827 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:43.257707 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:43.257617 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:43.357809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:43.357771 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:43.837032 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:43.836994 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:44.238735 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:44.238698 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:45.043390 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:45.043350 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:45.224421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:45.224370 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:45.278912 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:45.278876 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:45.433338 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:45.433304 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:45.846705 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:45.846600 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:46.236347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.236305 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:46.306924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.306887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:46.307417 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.306998 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:46.307417 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:46.307145 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 07:15:46.307417 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:46.307214 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls podName:2613fe87-dc24-45d8-8897-f211058dbd51 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:54.307198076 +0000 UTC m=+64.078648853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7fzh" (UID: "2613fe87-dc24-45d8-8897-f211058dbd51") : secret "insights-runtime-extractor-tls" not found Jul 29 07:15:46.307615 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.307599 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2613fe87-dc24-45d8-8897-f211058dbd51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:46.425470 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.425422 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:46.425629 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.425498 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:15:46.425861 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.425848 2568 scope.go:117] "RemoveContainer" containerID="7ee33eecb52ed2021fa3ef905a4cf9eefeada215c9c3721e5c0d24c6f0f5eb0e" Jul 29 07:15:46.426016 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:15:46.425999 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ttnwj_openshift-console-operator(7a9c8dc2-3c50-48e2-a097-7313bfbd4de3)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" podUID="7a9c8dc2-3c50-48e2-a097-7313bfbd4de3" Jul 29 07:15:46.630903 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.630811 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:46.848967 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.848915 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:46.918185 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:46.918150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:47.280665 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:47.280571 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:47.674011 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:47.673976 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:47.732887 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:47.732852 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:47.832057 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:47.832019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:48.458837 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:48.458805 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:48.559600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:48.559563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:48.828869 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:48.828765 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:49.231123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:49.231088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:49.332149 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:49.332121 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:49.439820 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:49.439785 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:49.828527 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:49.828489 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:50.457608 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:50.457575 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:50.840578 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:50.840480 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:51.252369 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:51.252326 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:51.432058 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:51.432014 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:51.837011 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:51.836974 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:51.937732 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:51.937704 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrthw" Jul 29 07:15:52.675958 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:52.675926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:52.845955 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:52.845918 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:53.059899 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:53.059810 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:53.439703 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:53.439663 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:53.635030 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:53.634986 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:53.696347 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:53.696259 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:54.383268 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:54.383232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:54.385736 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:54.385712 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2613fe87-dc24-45d8-8897-f211058dbd51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7fzh\" (UID: \"2613fe87-dc24-45d8-8897-f211058dbd51\") " pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:54.450807 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:54.450769 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zj4vm\"" Jul 29 07:15:54.459049 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:54.459020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7fzh" Jul 29 07:15:54.596733 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:54.596694 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7fzh"] Jul 29 07:15:54.599745 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:54.599697 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2613fe87_dc24_45d8_8897_f211058dbd51.slice/crio-55d160e89725fb758a8ff70e6656c5027ec8655ad885593c661bd792e33f8c23 WatchSource:0}: Error finding container 55d160e89725fb758a8ff70e6656c5027ec8655ad885593c661bd792e33f8c23: Status 404 returned error can't find the container with id 55d160e89725fb758a8ff70e6656c5027ec8655ad885593c661bd792e33f8c23 Jul 29 07:15:55.153126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:55.152996 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7fzh" event={"ID":"2613fe87-dc24-45d8-8897-f211058dbd51","Type":"ContainerStarted","Data":"55d160e89725fb758a8ff70e6656c5027ec8655ad885593c661bd792e33f8c23"} Jul 29 07:15:56.063802 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.063767 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:56.115793 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.115757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:56.506212 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.506167 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:56.506421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.506292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:56.506421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.506322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:56.509240 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.509199 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrvhh\" (UniqueName: \"kubernetes.io/projected/4221d142-a9c3-4f22-8a63-216d86b24462-kube-api-access-nrvhh\") pod \"network-check-target-tgb4r\" (UID: \"4221d142-a9c3-4f22-8a63-216d86b24462\") " pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:56.509407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.509289 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 07:15:56.509407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.509294 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 07:15:56.519631 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.519603 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c71bd361-1cda-4ccd-8daf-3fc786c6d773-original-pull-secret\") pod \"global-pull-secret-syncer-82z89\" (UID: \"c71bd361-1cda-4ccd-8daf-3fc786c6d773\") " pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:56.519786 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.519723 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24546c0d-8683-4aac-8dd8-2f077292bc4f-metrics-certs\") pod \"network-metrics-daemon-df8cd\" (UID: \"24546c0d-8683-4aac-8dd8-2f077292bc4f\") " pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:56.750394 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.750361 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb4zn\"" Jul 29 07:15:56.757746 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.757668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df8cd" Jul 29 07:15:56.760900 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.760872 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rkrh4\"" Jul 29 07:15:56.764052 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.764007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z89" Jul 29 07:15:56.768283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.768255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:56.974371 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.972978 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-df8cd"] Jul 29 07:15:56.976524 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:56.976488 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24546c0d_8683_4aac_8dd8_2f077292bc4f.slice/crio-8070003f93154af83ad4e0d0ff39e61ae9d414272a6b093dc99f4c1186222825 WatchSource:0}: Error finding container 8070003f93154af83ad4e0d0ff39e61ae9d414272a6b093dc99f4c1186222825: Status 404 returned error can't find the container with id 8070003f93154af83ad4e0d0ff39e61ae9d414272a6b093dc99f4c1186222825 Jul 29 07:15:56.997267 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:56.997222 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82z89"] Jul 29 07:15:57.002334 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:57.002296 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71bd361_1cda_4ccd_8daf_3fc786c6d773.slice/crio-9e6fc1decd8eb3a390245e70fed0932b5ad11455db696269d8e8c7efda6c2722 WatchSource:0}: Error finding container 9e6fc1decd8eb3a390245e70fed0932b5ad11455db696269d8e8c7efda6c2722: Status 404 returned error can't find the container with id 9e6fc1decd8eb3a390245e70fed0932b5ad11455db696269d8e8c7efda6c2722 Jul 29 07:15:57.020449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.020411 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgb4r"] Jul 29 07:15:57.024021 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:15:57.023990 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4221d142_a9c3_4f22_8a63_216d86b24462.slice/crio-f3a0e2e63029c4c9457b87a06c95bc667c95f54593ccc3ac8127c8b9bd0d4b6f WatchSource:0}: Error finding container f3a0e2e63029c4c9457b87a06c95bc667c95f54593ccc3ac8127c8b9bd0d4b6f: Status 404 returned error can't find the container with id f3a0e2e63029c4c9457b87a06c95bc667c95f54593ccc3ac8127c8b9bd0d4b6f Jul 29 07:15:57.161852 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.161810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-df8cd" event={"ID":"24546c0d-8683-4aac-8dd8-2f077292bc4f","Type":"ContainerStarted","Data":"8070003f93154af83ad4e0d0ff39e61ae9d414272a6b093dc99f4c1186222825"} Jul 29 07:15:57.163601 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.163563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgb4r" event={"ID":"4221d142-a9c3-4f22-8a63-216d86b24462","Type":"ContainerStarted","Data":"335899333c101f521d7d9e73204442fe55b18f8a294ed628cc7aca8c95832768"} Jul 29 07:15:57.163768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.163624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgb4r" event={"ID":"4221d142-a9c3-4f22-8a63-216d86b24462","Type":"ContainerStarted","Data":"f3a0e2e63029c4c9457b87a06c95bc667c95f54593ccc3ac8127c8b9bd0d4b6f"} Jul 29 07:15:57.163768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.163669 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:15:57.165129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.165100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82z89" event={"ID":"c71bd361-1cda-4ccd-8daf-3fc786c6d773","Type":"ContainerStarted","Data":"9e6fc1decd8eb3a390245e70fed0932b5ad11455db696269d8e8c7efda6c2722"} Jul 29 07:15:57.179923 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:57.179871 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgb4r" podStartSLOduration=66.179851957 podStartE2EDuration="1m6.179851957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:57.178746205 +0000 UTC m=+66.950197006" watchObservedRunningTime="2026-07-29 07:15:57.179851957 +0000 UTC m=+66.951302755" Jul 29 07:15:58.021221 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.020982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:58.023929 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.023895 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"image-registry-684dc586b6-4cddw\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:58.122370 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.122331 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:58.122588 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.122393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:58.122588 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.122432 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:58.122588 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.122464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:58.122588 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.122500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:58.122588 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.122560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:58.123522 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.123491 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-service-ca-bundle\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:58.125541 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.125503 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eac22f6-6f8c-499c-ac0f-efb9459915e6-metrics-tls\") pod \"dns-default-xkkwv\" (UID: \"2eac22f6-6f8c-499c-ac0f-efb9459915e6\") " pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:58.125689 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.125577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7-metrics-certs\") pod \"router-default-6549448c57-9mrc9\" (UID: \"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7\") " pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:58.125689 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.125598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac2baf2-1488-4dc6-baec-f6f313f330e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lqljb\" (UID: \"1ac2baf2-1488-4dc6-baec-f6f313f330e0\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:58.126353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.126320 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7154b5ee-985a-4b27-96e7-34cd469b9b69-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sqbgr\" (UID: \"7154b5ee-985a-4b27-96e7-34cd469b9b69\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:58.127297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.127269 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6e776c2-47be-45d2-9a2e-f51c83917c11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wk92\" (UID: \"b6e776c2-47be-45d2-9a2e-f51c83917c11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:58.211636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.211602 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-846cg\"" Jul 29 07:15:58.218538 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.218504 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:15:58.269273 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.269227 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8vmfp\"" Jul 29 07:15:58.274447 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.274368 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rv5xs\"" Jul 29 07:15:58.276353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.276322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" Jul 29 07:15:58.281490 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.281460 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r8m8t\"" Jul 29 07:15:58.281770 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.281749 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" Jul 29 07:15:58.286207 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.286178 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6q689\"" Jul 29 07:15:58.289515 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.289489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:15:58.294415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.294388 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkkwv" Jul 29 07:15:58.304076 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.304024 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7wclq\"" Jul 29 07:15:58.311352 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:58.311315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" Jul 29 07:15:59.032466 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:59.032423 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:59.087098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:59.087066 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:59.693472 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:59.693441 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-684dc586b6-4cddw"] Jul 29 07:15:59.732301 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:15:59.732271 2568 scope.go:117] "RemoveContainer" containerID="7ee33eecb52ed2021fa3ef905a4cf9eefeada215c9c3721e5c0d24c6f0f5eb0e" Jul 29 07:16:00.198604 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:16:00.198559 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8bd8a7_0dd0_4ba5_9fa6_257c61a65a78.slice/crio-cfb7162c681749e17ed9d30a14981416b50ccb5d795cf7880668701fd29196c0 WatchSource:0}: Error finding container cfb7162c681749e17ed9d30a14981416b50ccb5d795cf7880668701fd29196c0: Status 404 returned error can't find the container with id cfb7162c681749e17ed9d30a14981416b50ccb5d795cf7880668701fd29196c0 Jul 29 07:16:00.472006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:00.471973 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6549448c57-9mrc9"] Jul 29 07:16:00.490882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:00.490806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xkkwv"] Jul 29 07:16:00.498704 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:16:00.496558 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eac22f6_6f8c_499c_ac0f_efb9459915e6.slice/crio-c98c0b87e04069720f47c4f1639fa8bd449e0b8402058512815dff5836c8136b WatchSource:0}: Error finding container c98c0b87e04069720f47c4f1639fa8bd449e0b8402058512815dff5836c8136b: Status 404 returned error can't find the container with id c98c0b87e04069720f47c4f1639fa8bd449e0b8402058512815dff5836c8136b Jul 29 07:16:00.500180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:00.500139 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr"] Jul 29 07:16:00.527916 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:00.527877 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb"] Jul 29 07:16:00.539263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:00.539228 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92"] Jul 29 07:16:01.193492 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.193452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7fzh" event={"ID":"2613fe87-dc24-45d8-8897-f211058dbd51","Type":"ContainerStarted","Data":"0db527445d8fd9b3c4a931ffcabef54852b32f02d313e57c130b6199aad27b39"} Jul 29 07:16:01.195545 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.195506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" event={"ID":"b6e776c2-47be-45d2-9a2e-f51c83917c11","Type":"ContainerStarted","Data":"291fe0d0d3b53d82f5ef7cb5875ad32c6cd08e1cc64ed6985a6fdeebb9043295"} Jul 29 07:16:01.198683 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.198647 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkkwv" event={"ID":"2eac22f6-6f8c-499c-ac0f-efb9459915e6","Type":"ContainerStarted","Data":"c98c0b87e04069720f47c4f1639fa8bd449e0b8402058512815dff5836c8136b"} Jul 29 07:16:01.205010 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.204976 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:16:01.205196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.205111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" event={"ID":"7a9c8dc2-3c50-48e2-a097-7313bfbd4de3","Type":"ContainerStarted","Data":"d51bb1876996ce05709dc018b6efd8ac923114e975692394b934651a17490fe0"} Jul 29 07:16:01.207303 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.207254 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:16:01.214281 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.214226 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" Jul 29 07:16:01.216072 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.215978 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-df8cd" event={"ID":"24546c0d-8683-4aac-8dd8-2f077292bc4f","Type":"ContainerStarted","Data":"b2379533484b9e6f016e712605e739ca41f2df99f74776c9126bf3505dfd2138"} Jul 29 07:16:01.216287 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.216240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-df8cd" event={"ID":"24546c0d-8683-4aac-8dd8-2f077292bc4f","Type":"ContainerStarted","Data":"449b9497db2ee242421124b581b07d8744452443753231101249b76d98d8a612"} Jul 29 07:16:01.219479 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.219395 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" event={"ID":"1ac2baf2-1488-4dc6-baec-f6f313f330e0","Type":"ContainerStarted","Data":"8eac705461576fc1eb55ec39a6ae6691b5a8675c7ba94aee1018caa5984d4546"} Jul 29 07:16:01.222417 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.222385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" event={"ID":"7154b5ee-985a-4b27-96e7-34cd469b9b69","Type":"ContainerStarted","Data":"568c4c0ca3559dc4ae93b4ded6296e742ea71a1bf0f968bc52de450867a0b2a3"} Jul 29 07:16:01.224766 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.224707 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-ttnwj" podStartSLOduration=41.209228214 podStartE2EDuration="51.224688153s" totalImagesPullingTime="10.015459939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:01.224505095 +0000 UTC m=+70.995955897" watchObservedRunningTime="2026-07-29 07:16:01.224688153 +0000 UTC m=+70.996138953" Jul 29 07:16:01.235409 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.234614 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" event={"ID":"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78","Type":"ContainerStarted","Data":"989716c09a9b9de6d3f44d35ec604afa275c692eebe48125775180cff2b7f80c"} Jul 29 07:16:01.235409 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.234678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" event={"ID":"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78","Type":"ContainerStarted","Data":"cfb7162c681749e17ed9d30a14981416b50ccb5d795cf7880668701fd29196c0"} Jul 29 07:16:01.235409 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.235346 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:16:01.239475 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.238795 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6549448c57-9mrc9" event={"ID":"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7","Type":"ContainerStarted","Data":"fe74da8a1aa737fe633a6b12da9273d613f5ee19e27c70fe2cc2499e28c3bbed"} Jul 29 07:16:01.239475 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.238855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6549448c57-9mrc9" event={"ID":"1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7","Type":"ContainerStarted","Data":"8b1bf858e53c5543d91125f6b6e73f40f9aca4feea89d88d258f647d0c709693"} Jul 29 07:16:01.250482 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.250416 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-df8cd" podStartSLOduration=66.969282431 podStartE2EDuration="1m10.250393873s" totalImagesPullingTime="3.281111442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:01.250097036 +0000 UTC m=+71.021547835" watchObservedRunningTime="2026-07-29 07:16:01.250393873 +0000 UTC m=+71.021844673" Jul 29 07:16:01.291103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.290998 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:16:01.294815 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.294783 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:16:01.297318 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.297269 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:01.314798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.314726 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" podStartSLOduration=70.314701346 podStartE2EDuration="1m10.314701346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:01.314325506 +0000 UTC m=+71.085776305" watchObservedRunningTime="2026-07-29 07:16:01.314701346 +0000 UTC m=+71.086152147" Jul 29 07:16:01.341730 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.341649 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6549448c57-9mrc9" podStartSLOduration=51.341609973 podStartE2EDuration="51.341609973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:01.339928694 +0000 UTC m=+71.111379493" watchObservedRunningTime="2026-07-29 07:16:01.341609973 +0000 UTC m=+71.113060777" Jul 29 07:16:01.410595 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.410558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:01.467746 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.467648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:01.619460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.619404 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:01.944752 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:01.944669 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:02.036319 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.036277 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:02.198663 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.197131 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh"] Jul 29 07:16:02.219311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.211111 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt"] Jul 29 07:16:02.219311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.212024 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.219311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.215378 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 29 07:16:02.219311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.215650 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 07:16:02.219311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.215854 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-b4gjs\"" Jul 29 07:16:02.234300 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.220515 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 07:16:02.234300 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.226839 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 07:16:02.234775 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.234310 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m"] Jul 29 07:16:02.237431 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.236168 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.240259 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240230 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 07:16:02.240538 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240517 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 07:16:02.240637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240590 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m"] Jul 29 07:16:02.240637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240628 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh"] Jul 29 07:16:02.240736 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt"] Jul 29 07:16:02.240736 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240715 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 07:16:02.240836 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240750 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.240922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.240903 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 07:16:02.243591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.243359 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 07:16:02.250912 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.250883 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:16:02.253724 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.252782 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6549448c57-9mrc9" Jul 29 07:16:02.274224 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.274187 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/323623d3-a99b-478a-a7d7-6dcb90924bac-tmp\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.274833 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.274809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9r2\" (UniqueName: \"kubernetes.io/projected/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-kube-api-access-sq9r2\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.275058 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.275222 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-hub\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.275337 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275323 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crk5q\" (UniqueName: \"kubernetes.io/projected/323623d3-a99b-478a-a7d7-6dcb90924bac-kube-api-access-crk5q\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.275439 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.275628 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275614 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d858d2f-b713-43af-aeb3-cc8a2390563b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cbfdffddb-q26kh\" (UID: \"9d858d2f-b713-43af-aeb3-cc8a2390563b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.275739 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7wz\" (UniqueName: \"kubernetes.io/projected/9d858d2f-b713-43af-aeb3-cc8a2390563b-kube-api-access-sc7wz\") pod \"managed-serviceaccount-addon-agent-7cbfdffddb-q26kh\" (UID: \"9d858d2f-b713-43af-aeb3-cc8a2390563b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.275849 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/323623d3-a99b-478a-a7d7-6dcb90924bac-klusterlet-config\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.276002 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.275989 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-ca\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.276127 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.276112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.281962 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.281930 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-ldld4"] Jul 29 07:16:02.285149 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.285123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:02.288442 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.288393 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:02.289088 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.288728 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 07:16:02.289088 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.288982 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vnsvx\"" Jul 29 07:16:02.289584 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.289372 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 07:16:02.303973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.303933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-ldld4"] Jul 29 07:16:02.376827 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.376784 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d858d2f-b713-43af-aeb3-cc8a2390563b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cbfdffddb-q26kh\" (UID: \"9d858d2f-b713-43af-aeb3-cc8a2390563b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.376998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.376842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7wz\" (UniqueName: \"kubernetes.io/projected/9d858d2f-b713-43af-aeb3-cc8a2390563b-kube-api-access-sc7wz\") pod \"managed-serviceaccount-addon-agent-7cbfdffddb-q26kh\" (UID: \"9d858d2f-b713-43af-aeb3-cc8a2390563b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.376998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.376878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/323623d3-a99b-478a-a7d7-6dcb90924bac-klusterlet-config\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.376998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.376910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwd2\" (UniqueName: \"kubernetes.io/projected/e31b612f-6955-4087-a735-b5e82e5dbb83-kube-api-access-cfwd2\") pod \"downloads-7c97dc4555-ldld4\" (UID: \"e31b612f-6955-4087-a735-b5e82e5dbb83\") " pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:02.376998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.376970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-ca\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.376998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.376993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.377315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.377089 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/323623d3-a99b-478a-a7d7-6dcb90924bac-tmp\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.377315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.377116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9r2\" (UniqueName: \"kubernetes.io/projected/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-kube-api-access-sq9r2\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.377315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.377156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.377315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.377193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-hub\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.377315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.377273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crk5q\" (UniqueName: \"kubernetes.io/projected/323623d3-a99b-478a-a7d7-6dcb90924bac-kube-api-access-crk5q\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.377315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.377307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.378625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.378594 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/323623d3-a99b-478a-a7d7-6dcb90924bac-tmp\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.381095 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.380940 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.386325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.386261 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:02.387709 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.387645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.396063 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.393736 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.401970 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.401845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crk5q\" (UniqueName: \"kubernetes.io/projected/323623d3-a99b-478a-a7d7-6dcb90924bac-kube-api-access-crk5q\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.402607 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.402576 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7wz\" (UniqueName: \"kubernetes.io/projected/9d858d2f-b713-43af-aeb3-cc8a2390563b-kube-api-access-sc7wz\") pod \"managed-serviceaccount-addon-agent-7cbfdffddb-q26kh\" (UID: \"9d858d2f-b713-43af-aeb3-cc8a2390563b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.404484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.403656 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-hub\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.404484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.404125 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-ca\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.404737 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.404715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/323623d3-a99b-478a-a7d7-6dcb90924bac-klusterlet-config\") pod \"klusterlet-addon-workmgr-59d69c67b6-sdq5m\" (UID: \"323623d3-a99b-478a-a7d7-6dcb90924bac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.405386 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.405351 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d858d2f-b713-43af-aeb3-cc8a2390563b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cbfdffddb-q26kh\" (UID: \"9d858d2f-b713-43af-aeb3-cc8a2390563b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.410297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.410222 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9r2\" (UniqueName: \"kubernetes.io/projected/69ac70d9-a5ce-414c-9db5-2d6f102c81f9-kube-api-access-sq9r2\") pod \"cluster-proxy-proxy-agent-78fb6fcd4-854kt\" (UID: \"69ac70d9-a5ce-414c-9db5-2d6f102c81f9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.480716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.480361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwd2\" (UniqueName: \"kubernetes.io/projected/e31b612f-6955-4087-a735-b5e82e5dbb83-kube-api-access-cfwd2\") pod \"downloads-7c97dc4555-ldld4\" (UID: \"e31b612f-6955-4087-a735-b5e82e5dbb83\") " pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:02.491384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.491248 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:02.492747 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.492719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwd2\" (UniqueName: \"kubernetes.io/projected/e31b612f-6955-4087-a735-b5e82e5dbb83-kube-api-access-cfwd2\") pod \"downloads-7c97dc4555-ldld4\" (UID: \"e31b612f-6955-4087-a735-b5e82e5dbb83\") " pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:02.545266 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.545218 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" Jul 29 07:16:02.574547 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.574512 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" Jul 29 07:16:02.575137 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.575110 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:02.588291 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.588236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:02.613194 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.613156 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:02.679275 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:02.679100 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:03.005395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:03.005354 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:03.152623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:03.152576 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:03.388640 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:03.388543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:04.365103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:04.365073 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:04.464844 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:04.464808 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:07.423412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:07.423372 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:07.759281 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:07.759135 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:07.858255 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:07.858218 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:08.120277 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:08.120241 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-ldld4"] Jul 29 07:16:08.295306 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:16:08.295231 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31b612f_6955_4087_a735_b5e82e5dbb83.slice/crio-5a6d025cdad505bae4a1a6028b8966b0e72de07429e6a9bf5b1b3f7e50144573 WatchSource:0}: Error finding container 5a6d025cdad505bae4a1a6028b8966b0e72de07429e6a9bf5b1b3f7e50144573: Status 404 returned error can't find the container with id 5a6d025cdad505bae4a1a6028b8966b0e72de07429e6a9bf5b1b3f7e50144573 Jul 29 07:16:09.266633 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.266555 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh"] Jul 29 07:16:09.300451 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.300127 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m"] Jul 29 07:16:09.302660 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.302351 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-ldld4" event={"ID":"e31b612f-6955-4087-a735-b5e82e5dbb83","Type":"ContainerStarted","Data":"5a6d025cdad505bae4a1a6028b8966b0e72de07429e6a9bf5b1b3f7e50144573"} Jul 29 07:16:09.305062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.305014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" event={"ID":"b6e776c2-47be-45d2-9a2e-f51c83917c11","Type":"ContainerStarted","Data":"a375fef9cc11830b3ba96f1681e8d407995821c4a9eaa8e9fb8d0c5c01fdfd04"} Jul 29 07:16:09.308664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.308199 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" event={"ID":"1ac2baf2-1488-4dc6-baec-f6f313f330e0","Type":"ContainerStarted","Data":"d964b2419028b7bb46f676359e6fcc730c38439a1d9a2b0dcb52eebf4445c6aa"} Jul 29 07:16:09.311335 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.311302 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" event={"ID":"7154b5ee-985a-4b27-96e7-34cd469b9b69","Type":"ContainerStarted","Data":"a7b38cb42bceab4f96f75e8b8caa925f2233f25e659b064e24f2caa035cd111e"} Jul 29 07:16:09.328968 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.328906 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lqljb" podStartSLOduration=50.918771709 podStartE2EDuration="59.328881776s" totalImagesPullingTime="8.410110067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:09.32530842 +0000 UTC m=+79.096759219" watchObservedRunningTime="2026-07-29 07:16:09.328881776 +0000 UTC m=+79.100332575" Jul 29 07:16:09.330290 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.330249 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt"] Jul 29 07:16:09.348613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.348510 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sqbgr" podStartSLOduration=43.782884496 podStartE2EDuration="52.348486434s" totalImagesPullingTime="8.565601938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:09.346633787 +0000 UTC m=+79.118084587" watchObservedRunningTime="2026-07-29 07:16:09.348486434 +0000 UTC m=+79.119937250" Jul 29 07:16:09.524685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.524590 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:09.620613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.620514 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:09.764372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:09.764285 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:10.101519 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.101376 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:10.325760 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.325648 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" event={"ID":"9d858d2f-b713-43af-aeb3-cc8a2390563b","Type":"ContainerStarted","Data":"cf759269315e713afdd3bdaf05fad5f4c269f2fd9e24ed1d12e3121690f58457"} Jul 29 07:16:10.331603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.331548 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82z89" event={"ID":"c71bd361-1cda-4ccd-8daf-3fc786c6d773","Type":"ContainerStarted","Data":"7ff9bf33bfa3054655598ea57e2ce703d8f6001b3147dfacf423cff1fa822ef8"} Jul 29 07:16:10.334581 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.334488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7fzh" event={"ID":"2613fe87-dc24-45d8-8897-f211058dbd51","Type":"ContainerStarted","Data":"57899e16129725bc7ab880341f81393ca53cf75637f8348882975a6732cce727"} Jul 29 07:16:10.342613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.341927 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" event={"ID":"b6e776c2-47be-45d2-9a2e-f51c83917c11","Type":"ContainerStarted","Data":"738075a5f4bc9903cd0984d2a4f74fba6e25253e9cbf200e96a865652299ab81"} Jul 29 07:16:10.348688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.347879 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkkwv" event={"ID":"2eac22f6-6f8c-499c-ac0f-efb9459915e6","Type":"ContainerStarted","Data":"f03cfb43cb8328ea697c3b74477afb049d6af161d801b7790513be315bb6b9ca"} Jul 29 07:16:10.348688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.347949 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkkwv" event={"ID":"2eac22f6-6f8c-499c-ac0f-efb9459915e6","Type":"ContainerStarted","Data":"34a9f18e254644185707814f622435f8ac3c1bb7ba6d983bba6ad739f2ae5012"} Jul 29 07:16:10.348688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.348619 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xkkwv" Jul 29 07:16:10.364126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.362318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" event={"ID":"69ac70d9-a5ce-414c-9db5-2d6f102c81f9","Type":"ContainerStarted","Data":"0cf7e4a2ee45e932e0dc343fcde2ec2845941395aa905c7a9f206c8ef8c4c7b5"} Jul 29 07:16:10.367956 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.367920 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" event={"ID":"323623d3-a99b-478a-a7d7-6dcb90924bac","Type":"ContainerStarted","Data":"c2000a6ba8a0f4f00b642897fc229e8bbf0a3e4046275687fa36b612f6d26934"} Jul 29 07:16:10.376095 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.375055 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-82z89" podStartSLOduration=67.30628592 podStartE2EDuration="1m19.375010518s" totalImagesPullingTime="12.068724598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:10.352919754 +0000 UTC m=+80.124370555" watchObservedRunningTime="2026-07-29 07:16:10.375010518 +0000 UTC m=+80.146461322" Jul 29 07:16:10.408890 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:10.407379 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xkkwv" podStartSLOduration=35.832859844 podStartE2EDuration="44.407362118s" totalImagesPullingTime="8.574502274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:10.406945718 +0000 UTC m=+80.178396519" watchObservedRunningTime="2026-07-29 07:16:10.407362118 +0000 UTC m=+80.178812916" Jul 29 07:16:11.065659 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:11.065598 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:11.132833 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:11.132690 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:12.974502 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:12.974416 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:13.041763 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:13.041676 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:13.464052 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:13.463918 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:13.524665 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:13.524554 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:14.428722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:14.428676 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:14.519635 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:14.519595 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:15.398602 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:15.398475 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:15.474914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:15.474805 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:16.578157 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:16.578078 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:16.642517 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:16.642434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:16.808227 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:16.808142 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:16.899534 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:16.899450 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:17.131231 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:17.131193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:18.225520 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:18.225473 2568 patch_prober.go:28] interesting pod/image-registry-684dc586b6-4cddw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 07:16:18.225988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:18.225556 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:16:19.137987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.137954 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:19.200424 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.200394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:19.262622 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.262426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:19.425066 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.425009 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7fzh" event={"ID":"2613fe87-dc24-45d8-8897-f211058dbd51","Type":"ContainerStarted","Data":"94951664abf07e479b35f4e6b7cfb5c4d2b45e6e432ba5478017316b0f8a775f"} Jul 29 07:16:19.427527 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.427482 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" event={"ID":"69ac70d9-a5ce-414c-9db5-2d6f102c81f9","Type":"ContainerStarted","Data":"388ee2bd9d68d0927526b1e93ac392d23b35d566310df22841ab45819f3fe947"} Jul 29 07:16:19.427696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.427539 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" event={"ID":"69ac70d9-a5ce-414c-9db5-2d6f102c81f9","Type":"ContainerStarted","Data":"7b4b85aae7828028b201e12a38312434fe76fe328f178016df55fe9d82243524"} Jul 29 07:16:19.427696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.427557 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" event={"ID":"69ac70d9-a5ce-414c-9db5-2d6f102c81f9","Type":"ContainerStarted","Data":"5cfa0a8e157467a8fd28b06a7684e17cf0484abc00635f3c587de89357c0d748"} Jul 29 07:16:19.429897 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.429864 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" event={"ID":"323623d3-a99b-478a-a7d7-6dcb90924bac","Type":"ContainerStarted","Data":"94f9a2264613d111ea73084dfd6a615da1e969b5ff661f99cee243360fcac1d8"} Jul 29 07:16:19.430097 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.430076 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:19.431808 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.431755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" event={"ID":"9d858d2f-b713-43af-aeb3-cc8a2390563b","Type":"ContainerStarted","Data":"51acf1f9ff72a3c4662cad933c5bf18d09414a7a5bcb2047750dff5d59ea5141"} Jul 29 07:16:19.432176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.432151 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" Jul 29 07:16:19.442897 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.442836 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wk92" podStartSLOduration=60.971379214 podStartE2EDuration="1m9.44281799s" totalImagesPullingTime="8.471438776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:10.427157388 +0000 UTC m=+80.198608214" watchObservedRunningTime="2026-07-29 07:16:19.44281799 +0000 UTC m=+89.214268793" Jul 29 07:16:19.444130 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.444087 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x7fzh" podStartSLOduration=17.27595396 podStartE2EDuration="41.4440698s" totalImagesPullingTime="24.16811584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:19.442639339 +0000 UTC m=+89.214090138" watchObservedRunningTime="2026-07-29 07:16:19.4440698 +0000 UTC m=+89.215520592" Jul 29 07:16:19.460229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.460158 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cbfdffddb-q26kh" podStartSLOduration=7.759625836 podStartE2EDuration="17.460134447s" totalImagesPullingTime="9.700508611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:19.458855775 +0000 UTC m=+89.230306574" watchObservedRunningTime="2026-07-29 07:16:19.460134447 +0000 UTC m=+89.231585247" Jul 29 07:16:19.476114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.475951 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59d69c67b6-sdq5m" podStartSLOduration=7.752030734 podStartE2EDuration="17.475931288s" totalImagesPullingTime="9.723900554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:19.47568456 +0000 UTC m=+89.247135351" watchObservedRunningTime="2026-07-29 07:16:19.475931288 +0000 UTC m=+89.247382088" Jul 29 07:16:19.494860 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:19.494788 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb6fcd4-854kt" podStartSLOduration=7.774748137 podStartE2EDuration="17.4947637s" totalImagesPullingTime="9.720015563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:19.494005582 +0000 UTC m=+89.265456385" watchObservedRunningTime="2026-07-29 07:16:19.4947637 +0000 UTC m=+89.266214505" Jul 29 07:16:20.185963 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:20.185929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:20.352082 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:20.352027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:20.455842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:20.455744 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:21.387437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:21.387390 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:21.389025 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:21.389004 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xkkwv" Jul 29 07:16:23.260950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:23.260911 2568 patch_prober.go:28] interesting pod/image-registry-684dc586b6-4cddw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 07:16:23.261377 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:23.260976 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:16:23.961284 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:23.961242 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.052892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.052853 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.173572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.173532 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.245059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.244952 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lw92m"] Jul 29 07:16:24.266929 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.266765 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.302023 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.301000 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.304846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.304220 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 07:16:24.304846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.304538 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ph7j\"" Jul 29 07:16:24.304846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.304752 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 07:16:24.305112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.304925 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 07:16:24.305564 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.305549 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 07:16:24.341061 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.340948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.408505 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408465 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-root\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408690 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8lg\" (UniqueName: \"kubernetes.io/projected/80278e74-48be-4578-84fd-0956760475b2-kube-api-access-sh8lg\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408690 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408690 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-sys\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80278e74-48be-4578-84fd-0956760475b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80278e74-48be-4578-84fd-0956760475b2-metrics-client-ca\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.408854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-node-exporter-wtmp\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.409071 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.408856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80278e74-48be-4578-84fd-0956760475b2-node-exporter-textfile\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510068 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.509938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-node-exporter-wtmp\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510068 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.509996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80278e74-48be-4578-84fd-0956760475b2-node-exporter-textfile\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510068 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-root\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8lg\" (UniqueName: \"kubernetes.io/projected/80278e74-48be-4578-84fd-0956760475b2-kube-api-access-sh8lg\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510199 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-sys\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80278e74-48be-4578-84fd-0956760475b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80278e74-48be-4578-84fd-0956760475b2-metrics-client-ca\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510766 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-node-exporter-wtmp\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.510907 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.510886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80278e74-48be-4578-84fd-0956760475b2-metrics-client-ca\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.511016 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:16:24.511002 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 07:16:24.511087 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.511030 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80278e74-48be-4578-84fd-0956760475b2-node-exporter-textfile\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.511087 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:16:24.511084 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls podName:80278e74-48be-4578-84fd-0956760475b2 nodeName:}" failed. No retries permitted until 2026-07-29 07:16:25.011063693 +0000 UTC m=+94.782514486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls") pod "node-exporter-lw92m" (UID: "80278e74-48be-4578-84fd-0956760475b2") : secret "node-exporter-tls" not found Jul 29 07:16:24.511175 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.511128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-root\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.511175 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.511131 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80278e74-48be-4578-84fd-0956760475b2-sys\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.511602 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.511564 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80278e74-48be-4578-84fd-0956760475b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.523491 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.523461 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.536436 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.536375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8lg\" (UniqueName: \"kubernetes.io/projected/80278e74-48be-4578-84fd-0956760475b2-kube-api-access-sh8lg\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:24.610391 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.610075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.718173 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.718122 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.810924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.810801 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:24.911609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:24.911558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:25.015986 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.015949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:25.016188 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:16:25.016147 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 07:16:25.016256 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:16:25.016233 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls podName:80278e74-48be-4578-84fd-0956760475b2 nodeName:}" failed. No retries permitted until 2026-07-29 07:16:26.016209851 +0000 UTC m=+95.787660633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls") pod "node-exporter-lw92m" (UID: "80278e74-48be-4578-84fd-0956760475b2") : secret "node-exporter-tls" not found Jul 29 07:16:25.152957 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.152926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:25.252912 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.252872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:25.329527 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.329484 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:25.468998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.468879 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:25.777759 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.777664 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:25.832214 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:25.832132 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:26.026514 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:26.026468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:26.029473 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:26.029405 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80278e74-48be-4578-84fd-0956760475b2-node-exporter-tls\") pod \"node-exporter-lw92m\" (UID: \"80278e74-48be-4578-84fd-0956760475b2\") " pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:26.074117 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:26.074067 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:26.123652 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:26.123604 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lw92m" Jul 29 07:16:26.141379 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:26.141283 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:27.173549 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:27.173513 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:27.274482 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:27.274441 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:27.383728 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:27.383693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:27.475279 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:27.475190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:27.575501 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:27.575456 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:27.675967 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:27.675848 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:28.023219 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:28.023115 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:28.173638 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:28.173606 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgb4r" Jul 29 07:16:28.223706 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:28.223653 2568 patch_prober.go:28] interesting pod/image-registry-684dc586b6-4cddw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 07:16:28.223895 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:28.223713 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:16:29.012949 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.012832 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:29.089786 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.089730 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:29.308010 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.307065 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-684dc586b6-4cddw"] Jul 29 07:16:29.313700 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.313661 2568 patch_prober.go:28] interesting pod/image-registry-684dc586b6-4cddw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 07:16:29.313892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.313737 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:16:29.395860 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.395809 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:29.473200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:29.473163 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:29.711804 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:16:29.711712 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80278e74_48be_4578_84fd_0956760475b2.slice/crio-b35ecb159140fb3a7f456a4eaaaa20f4e5d1019a57e0f590037ae93e0ed6e0e6 WatchSource:0}: Error finding container b35ecb159140fb3a7f456a4eaaaa20f4e5d1019a57e0f590037ae93e0ed6e0e6: Status 404 returned error can't find the container with id b35ecb159140fb3a7f456a4eaaaa20f4e5d1019a57e0f590037ae93e0ed6e0e6 Jul 29 07:16:30.487246 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.487193 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw92m" event={"ID":"80278e74-48be-4578-84fd-0956760475b2","Type":"ContainerStarted","Data":"b35ecb159140fb3a7f456a4eaaaa20f4e5d1019a57e0f590037ae93e0ed6e0e6"} Jul 29 07:16:30.491504 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.491432 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-ldld4" event={"ID":"e31b612f-6955-4087-a735-b5e82e5dbb83","Type":"ContainerStarted","Data":"66a962b06df95d317a60b94f4db10febba49a3da35a2067fbdc7d2218f47f4b3"} Jul 29 07:16:30.492779 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.492229 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:30.507902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.507868 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-ldld4" Jul 29 07:16:30.545083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.544984 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-ldld4" podStartSLOduration=7.058468602 podStartE2EDuration="28.544957963s" totalImagesPullingTime="21.486489361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:30.514656435 +0000 UTC m=+100.286107235" watchObservedRunningTime="2026-07-29 07:16:30.544957963 +0000 UTC m=+100.316408764" Jul 29 07:16:30.692182 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.692144 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:30.793062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:30.792962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:31.980711 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:31.980616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:32.083229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.083183 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:32.334297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.334201 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:32.460107 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.460067 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:32.509542 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.509495 2568 generic.go:356] "Generic (PLEG): container finished" podID="80278e74-48be-4578-84fd-0956760475b2" containerID="c6af0a0af03d0591badac7f8734545c88d508b01cf85903f7b27381ccacd4681" exitCode=0 Jul 29 07:16:32.509734 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.509588 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw92m" event={"ID":"80278e74-48be-4578-84fd-0956760475b2","Type":"ContainerDied","Data":"c6af0a0af03d0591badac7f8734545c88d508b01cf85903f7b27381ccacd4681"} Jul 29 07:16:32.677324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.677283 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:32.776181 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:32.776141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:33.518112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:33.518071 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw92m" event={"ID":"80278e74-48be-4578-84fd-0956760475b2","Type":"ContainerStarted","Data":"5a27543181d55b26ac23ed4570004e8834fb176a0577d8e1ee1827beb4bd0d2c"} Jul 29 07:16:33.518678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:33.518134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw92m" event={"ID":"80278e74-48be-4578-84fd-0956760475b2","Type":"ContainerStarted","Data":"639ae16557b0e1ace85dc2908e3bf8f9831d868e40e33a59cba2bc32d2c93868"} Jul 29 07:16:33.539345 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:33.539284 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lw92m" podStartSLOduration=7.705756611 podStartE2EDuration="9.539261771s" totalImagesPullingTime="1.723559261s" totalInitContainerRuntime="109.945899ms" isStatefulPod=true podCreationTimestamp="2026-07-29 07:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:33.539196026 +0000 UTC m=+103.310646837" watchObservedRunningTime="2026-07-29 07:16:33.539261771 +0000 UTC m=+103.310712571" Jul 29 07:16:35.396879 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:35.396825 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:39.313972 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:39.313937 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:16:39.608308 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:39.608219 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:39.705592 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:39.705555 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:41.754583 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:41.754542 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:41.810385 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:41.810343 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:43.567096 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:43.567031 2568 generic.go:356] "Generic (PLEG): container finished" podID="2abf6566-aa10-4aec-bf74-26f935f88cda" containerID="0da43505d39d740a9bbabb4863aa460e79e4a110a9e26be37f721b14a2e00420" exitCode=0 Jul 29 07:16:43.567096 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:43.567082 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" event={"ID":"2abf6566-aa10-4aec-bf74-26f935f88cda","Type":"ContainerDied","Data":"0da43505d39d740a9bbabb4863aa460e79e4a110a9e26be37f721b14a2e00420"} Jul 29 07:16:43.567550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:43.567491 2568 scope.go:117] "RemoveContainer" containerID="0da43505d39d740a9bbabb4863aa460e79e4a110a9e26be37f721b14a2e00420" Jul 29 07:16:44.573646 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:44.573604 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-gwn5p" event={"ID":"2abf6566-aa10-4aec-bf74-26f935f88cda","Type":"ContainerStarted","Data":"efa9dea6e32b658f5e030469bb8aed730c6ee2a347e3cbc5ace5cb46c24be73f"} Jul 29 07:16:46.614963 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:46.614921 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:46.715661 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:46.715622 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:46.818255 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:46.818221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:54.331622 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.331559 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" containerID="cri-o://989716c09a9b9de6d3f44d35ec604afa275c692eebe48125775180cff2b7f80c" gracePeriod=30 Jul 29 07:16:54.438875 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.438838 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:54.492210 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.492173 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:54.614431 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.614394 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerID="989716c09a9b9de6d3f44d35ec604afa275c692eebe48125775180cff2b7f80c" exitCode=0 Jul 29 07:16:54.614589 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.614438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" event={"ID":"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78","Type":"ContainerDied","Data":"989716c09a9b9de6d3f44d35ec604afa275c692eebe48125775180cff2b7f80c"} Jul 29 07:16:54.614589 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.614495 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" event={"ID":"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78","Type":"ContainerDied","Data":"cfb7162c681749e17ed9d30a14981416b50ccb5d795cf7880668701fd29196c0"} Jul 29 07:16:54.614589 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.614509 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb7162c681749e17ed9d30a14981416b50ccb5d795cf7880668701fd29196c0" Jul 29 07:16:54.622986 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.622956 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:16:54.687892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.687849 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-bound-sa-token\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.687892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.687896 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-certificates\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.687919 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlb2b\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-kube-api-access-vlb2b\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.687940 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-installation-pull-secrets\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.687959 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-trusted-ca\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.687983 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-image-registry-private-configuration\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.688023 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-ca-trust-extracted\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688180 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.688111 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") pod \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\" (UID: \"3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78\") " Jul 29 07:16:54.688614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.688552 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:16:54.688899 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.688753 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:16:54.691059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.691005 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:16:54.691209 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.691001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:16:54.691268 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.691217 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-kube-api-access-vlb2b" (OuterVolumeSpecName: "kube-api-access-vlb2b") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "kube-api-access-vlb2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:16:54.691314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.691291 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:16:54.691366 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.691347 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:16:54.697059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.697009 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" (UID: "3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:16:54.788909 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788871 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-bound-sa-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788925 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-certificates\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788936 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlb2b\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-kube-api-access-vlb2b\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788945 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-installation-pull-secrets\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788955 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-trusted-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788964 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-image-registry-private-configuration\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788972 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-ca-trust-extracted\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:54.789123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:54.788980 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78-registry-tls\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:16:55.618024 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:55.617987 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684dc586b6-4cddw" Jul 29 07:16:55.637217 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:55.637170 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-684dc586b6-4cddw"] Jul 29 07:16:55.640064 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:55.640018 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-684dc586b6-4cddw"] Jul 29 07:16:56.060688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:56.060645 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:56.121407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:56.121360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:56.737292 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:16:56.737250 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" path="/var/lib/kubelet/pods/3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78/volumes" Jul 29 07:17:03.259194 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:03.259154 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:03.360725 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:03.360680 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:03.415235 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:03.415196 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:08.672561 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:08.672526 2568 generic.go:356] "Generic (PLEG): container finished" podID="602f7158-c486-4ca4-8be3-ca366799fe76" containerID="adac4ebb1f281617466c587b03a8971c2ab6c7626666534bdba2a6a110b9e693" exitCode=0 Jul 29 07:17:08.672991 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:08.672575 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" event={"ID":"602f7158-c486-4ca4-8be3-ca366799fe76","Type":"ContainerDied","Data":"adac4ebb1f281617466c587b03a8971c2ab6c7626666534bdba2a6a110b9e693"} Jul 29 07:17:08.672991 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:08.672901 2568 scope.go:117] "RemoveContainer" containerID="adac4ebb1f281617466c587b03a8971c2ab6c7626666534bdba2a6a110b9e693" Jul 29 07:17:09.678785 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:09.678744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jbm7k" event={"ID":"602f7158-c486-4ca4-8be3-ca366799fe76","Type":"ContainerStarted","Data":"1d84a601fd071b4d0e51d0e92b408039a597455e161def891d91dbe95f52a2ef"} Jul 29 07:17:13.697748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:13.697705 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2ee561b-df1e-47b7-a495-c6fee6888ec5" containerID="e6188278ce74df19283114846eccbb555a4f60d26174dd86c9e7e714cb52b80b" exitCode=0 Jul 29 07:17:13.698200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:13.697784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" event={"ID":"a2ee561b-df1e-47b7-a495-c6fee6888ec5","Type":"ContainerDied","Data":"e6188278ce74df19283114846eccbb555a4f60d26174dd86c9e7e714cb52b80b"} Jul 29 07:17:13.698200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:13.698155 2568 scope.go:117] "RemoveContainer" containerID="e6188278ce74df19283114846eccbb555a4f60d26174dd86c9e7e714cb52b80b" Jul 29 07:17:14.705105 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:14.705035 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp" event={"ID":"a2ee561b-df1e-47b7-a495-c6fee6888ec5","Type":"ContainerStarted","Data":"ba023e07fefb7139dd41652879a9d838c29c600013289831c3d40abf1b381e16"} Jul 29 07:17:40.390530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:40.390484 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:40.494625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:40.494585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:41.551121 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:41.549271 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:41.650475 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:41.650434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:52.275722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:52.275685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:52.997298 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:52.997254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:53.097782 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:53.097743 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:56.080544 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:56.080503 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:56.175791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:17:56.175752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:17.235437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:17.235390 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:17.900716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:17.900677 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:39.088644 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:39.088606 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:40.498635 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:40.498589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:42.666031 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:42.665990 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:42.718357 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:42.718317 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:44.276527 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:44.276492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:44.429895 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:44.429857 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:46.433259 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:46.433230 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:46.854664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:46.854567 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:49.258807 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:49.258775 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:49.317457 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:49.317426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:53.937421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:53.937386 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:54.458486 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:54.458455 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:56.551624 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:56.551587 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:56.650836 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:18:56.650797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:02.485422 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:02.485369 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:02.585524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:02.585488 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:04.664785 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:04.664740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:07.789233 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:07.789191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:14.544329 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:14.544235 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:18.140767 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:18.140728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:18.240484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:18.240435 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:18.745665 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:18.745627 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:18.984265 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:18.984219 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.079422 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.079327 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.218972 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.218931 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc"] Jul 29 07:19:19.219363 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.219254 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" Jul 29 07:19:19.219363 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.219266 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" Jul 29 07:19:19.219363 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.219334 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8bd8a7-0dd0-4ba5-9fa6-257c61a65a78" containerName="registry" Jul 29 07:19:19.221241 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.221224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.226760 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.226729 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 07:19:19.226949 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.226782 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sjl6m\"" Jul 29 07:19:19.227145 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.227124 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 07:19:19.227522 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.227506 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 07:19:19.228380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.228362 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 07:19:19.244928 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.244893 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc"] Jul 29 07:19:19.392555 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.392436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8b97b7a-9730-4448-8f5b-1823205df2b9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.392555 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.392491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8b97b7a-9730-4448-8f5b-1823205df2b9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.392789 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.392590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8b97b7a-9730-4448-8f5b-1823205df2b9-manifests\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.392789 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.392647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gs4n\" (UniqueName: \"kubernetes.io/projected/b8b97b7a-9730-4448-8f5b-1823205df2b9-kube-api-access-5gs4n\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.392789 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.392700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b97b7a-9730-4448-8f5b-1823205df2b9-tmp\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.400846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.400807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.486211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.486176 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.493767 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.493730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8b97b7a-9730-4448-8f5b-1823205df2b9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.493880 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.493785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8b97b7a-9730-4448-8f5b-1823205df2b9-manifests\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.493880 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.493824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gs4n\" (UniqueName: \"kubernetes.io/projected/b8b97b7a-9730-4448-8f5b-1823205df2b9-kube-api-access-5gs4n\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.493880 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.493846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b97b7a-9730-4448-8f5b-1823205df2b9-tmp\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.493880 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.493870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8b97b7a-9730-4448-8f5b-1823205df2b9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.494386 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.494356 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8b97b7a-9730-4448-8f5b-1823205df2b9-manifests\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.496446 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.496414 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b97b7a-9730-4448-8f5b-1823205df2b9-tmp\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.496683 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.496661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8b97b7a-9730-4448-8f5b-1823205df2b9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.496728 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.496692 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8b97b7a-9730-4448-8f5b-1823205df2b9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.505590 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.505558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gs4n\" (UniqueName: \"kubernetes.io/projected/b8b97b7a-9730-4448-8f5b-1823205df2b9-kube-api-access-5gs4n\") pod \"opendatahub-operator-controller-manager-6d6c966467-g9gpc\" (UID: \"b8b97b7a-9730-4448-8f5b-1823205df2b9\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.532430 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.532388 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:19.590842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.590800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.680642 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.680608 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc"] Jul 29 07:19:19.684886 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:19:19.684854 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b97b7a_9730_4448_8f5b_1823205df2b9.slice/crio-220fab36601317e72fc8a06a36a2984c7b6f99a38dbda6555729ac0e81766364 WatchSource:0}: Error finding container 220fab36601317e72fc8a06a36a2984c7b6f99a38dbda6555729ac0e81766364: Status 404 returned error can't find the container with id 220fab36601317e72fc8a06a36a2984c7b6f99a38dbda6555729ac0e81766364 Jul 29 07:19:19.685511 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.685487 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.752603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.752561 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.853274 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.853238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:19.954826 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:19.954736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:20.060723 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:20.060688 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:20.202182 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:20.202145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" event={"ID":"b8b97b7a-9730-4448-8f5b-1823205df2b9","Type":"ContainerStarted","Data":"220fab36601317e72fc8a06a36a2984c7b6f99a38dbda6555729ac0e81766364"} Jul 29 07:19:23.218945 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:23.218910 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" event={"ID":"b8b97b7a-9730-4448-8f5b-1823205df2b9","Type":"ContainerStarted","Data":"2ce6597feb1ea58463fa7fee14832503c10c56dbeeb81e7be6d9199aca9242f2"} Jul 29 07:19:23.630102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:23.630013 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:24.224792 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:24.224749 2568 generic.go:356] "Generic (PLEG): container finished" podID="b8b97b7a-9730-4448-8f5b-1823205df2b9" containerID="2ce6597feb1ea58463fa7fee14832503c10c56dbeeb81e7be6d9199aca9242f2" exitCode=0 Jul 29 07:19:24.225244 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:24.224847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" event={"ID":"b8b97b7a-9730-4448-8f5b-1823205df2b9","Type":"ContainerDied","Data":"2ce6597feb1ea58463fa7fee14832503c10c56dbeeb81e7be6d9199aca9242f2"} Jul 29 07:19:25.231684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:25.231652 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" event={"ID":"b8b97b7a-9730-4448-8f5b-1823205df2b9","Type":"ContainerStarted","Data":"c4bc08f6711ff07da394b2607fb35f8d0bc5e6679e628f81d73ba0a1671670ad"} Jul 29 07:19:25.232095 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:25.231749 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:25.255146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:25.255083 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" podStartSLOduration=1.6780961429999999 podStartE2EDuration="6.255060252s" totalImagesPullingTime="3.473890194s" totalInitContainerRuntime="1.103073915s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:25.253946797 +0000 UTC m=+275.025397600" watchObservedRunningTime="2026-07-29 07:19:25.255060252 +0000 UTC m=+275.026511044" Jul 29 07:19:31.113534 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:31.113500 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:31.212555 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:31.212518 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:36.238235 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:36.238203 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-g9gpc" Jul 29 07:19:38.462607 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.462569 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:38.685587 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.685546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:38.889886 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.889787 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:38.938957 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.938920 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg"] Jul 29 07:19:38.954758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.954730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:38.957636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.957611 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 07:19:38.958023 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.958005 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 07:19:38.972876 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:38.972846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg"] Jul 29 07:19:39.052924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.052885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tmp\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.052924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.052932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tls-certs\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.053220 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.053019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zk5\" (UniqueName: \"kubernetes.io/projected/9e6a3c00-100f-4c4e-a5dd-1908529e2269-kube-api-access-c9zk5\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.154092 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.154032 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tmp\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.154283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.154116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tls-certs\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.154283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.154172 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zk5\" (UniqueName: \"kubernetes.io/projected/9e6a3c00-100f-4c4e-a5dd-1908529e2269-kube-api-access-c9zk5\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.154283 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:39.154265 2568 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 29 07:19:39.154439 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:39.154330 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tls-certs podName:9e6a3c00-100f-4c4e-a5dd-1908529e2269 nodeName:}" failed. No retries permitted until 2026-07-29 07:19:39.654313941 +0000 UTC m=+289.425764718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tls-certs") pod "kube-auth-proxy-6dd888fd86-hl6vg" (UID: "9e6a3c00-100f-4c4e-a5dd-1908529e2269") : secret "kube-auth-proxy-tls" not found Jul 29 07:19:39.156616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.156595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tmp\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.165593 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.165563 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zk5\" (UniqueName: \"kubernetes.io/projected/9e6a3c00-100f-4c4e-a5dd-1908529e2269-kube-api-access-c9zk5\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.362149 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.362111 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dcg5q"] Jul 29 07:19:39.365578 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.365558 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:39.369831 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.369807 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 07:19:39.382544 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.382519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xb58s\"" Jul 29 07:19:39.399146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.399113 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dcg5q"] Jul 29 07:19:39.435176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.435083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:39.457154 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.457120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56b3b447-9444-4e22-ab2a-f38dc1244cb0-cert\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:39.457276 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.457163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdmx\" (UniqueName: \"kubernetes.io/projected/56b3b447-9444-4e22-ab2a-f38dc1244cb0-kube-api-access-vsdmx\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:39.525865 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.525830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:39.557671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.557633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56b3b447-9444-4e22-ab2a-f38dc1244cb0-cert\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:39.557671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.557678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdmx\" (UniqueName: \"kubernetes.io/projected/56b3b447-9444-4e22-ab2a-f38dc1244cb0-kube-api-access-vsdmx\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:39.557917 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:39.557805 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 07:19:39.557917 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:39.557888 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b3b447-9444-4e22-ab2a-f38dc1244cb0-cert podName:56b3b447-9444-4e22-ab2a-f38dc1244cb0 nodeName:}" failed. No retries permitted until 2026-07-29 07:19:40.057864319 +0000 UTC m=+289.829315099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/56b3b447-9444-4e22-ab2a-f38dc1244cb0-cert") pod "odh-model-controller-7998c67dc4-dcg5q" (UID: "56b3b447-9444-4e22-ab2a-f38dc1244cb0") : secret "odh-model-controller-webhook-cert" not found Jul 29 07:19:39.566691 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.566651 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdmx\" (UniqueName: \"kubernetes.io/projected/56b3b447-9444-4e22-ab2a-f38dc1244cb0-kube-api-access-vsdmx\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:39.626875 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.626840 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:39.658485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.658448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tls-certs\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.661062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.661026 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6a3c00-100f-4c4e-a5dd-1908529e2269-tls-certs\") pod \"kube-auth-proxy-6dd888fd86-hl6vg\" (UID: \"9e6a3c00-100f-4c4e-a5dd-1908529e2269\") " pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:39.727074 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.726955 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:39.864787 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:39.864755 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" Jul 29 07:19:40.035990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.035948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:40.036452 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.036428 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg"] Jul 29 07:19:40.038342 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:19:40.038317 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6a3c00_100f_4c4e_a5dd_1908529e2269.slice/crio-42c1f51a73069aee8f74fd70acefc1723f67b346ed364bb81cff378a4253b543 WatchSource:0}: Error finding container 42c1f51a73069aee8f74fd70acefc1723f67b346ed364bb81cff378a4253b543: Status 404 returned error can't find the container with id 42c1f51a73069aee8f74fd70acefc1723f67b346ed364bb81cff378a4253b543 Jul 29 07:19:40.062349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.062311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56b3b447-9444-4e22-ab2a-f38dc1244cb0-cert\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:40.065023 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.064997 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56b3b447-9444-4e22-ab2a-f38dc1244cb0-cert\") pod \"odh-model-controller-7998c67dc4-dcg5q\" (UID: \"56b3b447-9444-4e22-ab2a-f38dc1244cb0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:40.276276 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.276237 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:40.297548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.297501 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" event={"ID":"9e6a3c00-100f-4c4e-a5dd-1908529e2269","Type":"ContainerStarted","Data":"42c1f51a73069aee8f74fd70acefc1723f67b346ed364bb81cff378a4253b543"} Jul 29 07:19:40.393248 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.393207 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:40.394528 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.394500 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p2cn6"] Jul 29 07:19:40.399168 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.399148 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:40.402234 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.402208 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 07:19:40.420102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.414690 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p2cn6"] Jul 29 07:19:40.425704 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.425671 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx"] Jul 29 07:19:40.429251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.429229 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.433372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.433339 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 07:19:40.433599 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.433580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 07:19:40.434310 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.434290 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:19:40.434409 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.434291 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 07:19:40.434409 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.434338 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4dmts\"" Jul 29 07:19:40.442678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.442655 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dcg5q"] Jul 29 07:19:40.445714 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:19:40.445637 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b3b447_9444_4e22_ab2a_f38dc1244cb0.slice/crio-d8d923f13c951516f3947318f35a3b98795cb8dee35324d2eb5005e6e6bca61b WatchSource:0}: Error finding container d8d923f13c951516f3947318f35a3b98795cb8dee35324d2eb5005e6e6bca61b: Status 404 returned error can't find the container with id d8d923f13c951516f3947318f35a3b98795cb8dee35324d2eb5005e6e6bca61b Jul 29 07:19:40.449189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.449153 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx"] Jul 29 07:19:40.469732 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.469692 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899184eb-665d-4185-b920-29233d02f5e3-tls-certs\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:40.469917 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.469839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck5kn\" (UniqueName: \"kubernetes.io/projected/899184eb-665d-4185-b920-29233d02f5e3-kube-api-access-ck5kn\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:40.553154 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.553031 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:40.571114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571201 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899184eb-665d-4185-b920-29233d02f5e3-tls-certs\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:40.571349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc8k\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-kube-api-access-jgc8k\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767672-8156-4954-a189-dff457011f47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571301 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571510 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:40.571355 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 29 07:19:40.571510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571510 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:40.571408 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899184eb-665d-4185-b920-29233d02f5e3-tls-certs podName:899184eb-665d-4185-b920-29233d02f5e3 nodeName:}" failed. No retries permitted until 2026-07-29 07:19:41.071392178 +0000 UTC m=+290.842842955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/899184eb-665d-4185-b920-29233d02f5e3-tls-certs") pod "model-serving-api-54d89bd5fc-p2cn6" (UID: "899184eb-665d-4185-b920-29233d02f5e3") : secret "model-serving-api-tls" not found Jul 29 07:19:40.571510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571476 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c767672-8156-4954-a189-dff457011f47-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.571510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.571504 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck5kn\" (UniqueName: \"kubernetes.io/projected/899184eb-665d-4185-b920-29233d02f5e3-kube-api-access-ck5kn\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:40.580117 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.580086 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck5kn\" (UniqueName: \"kubernetes.io/projected/899184eb-665d-4185-b920-29233d02f5e3-kube-api-access-ck5kn\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:40.666769 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.666732 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:40.672422 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.672422 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.672667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc8k\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-kube-api-access-jgc8k\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.672667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767672-8156-4954-a189-dff457011f47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.672667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.672667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.672667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.672621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c767672-8156-4954-a189-dff457011f47-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.673194 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.673167 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767672-8156-4954-a189-dff457011f47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.675648 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.675624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c767672-8156-4954-a189-dff457011f47-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.675770 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.675623 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.675770 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.675708 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.676081 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.676058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.681092 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.681068 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.682099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.682075 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc8k\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-kube-api-access-jgc8k\") pod \"istiod-openshift-gateway-7b7995d5cd-2g6cx\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.738430 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.738401 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:40.910857 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:40.910828 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx"] Jul 29 07:19:40.912959 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:19:40.912926 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c767672_8156_4954_a189_dff457011f47.slice/crio-d71624a77ba203055d3305876db067875c67891faec274929a4c4b217ae94279 WatchSource:0}: Error finding container d71624a77ba203055d3305876db067875c67891faec274929a4c4b217ae94279: Status 404 returned error can't find the container with id d71624a77ba203055d3305876db067875c67891faec274929a4c4b217ae94279 Jul 29 07:19:41.075577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.075541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899184eb-665d-4185-b920-29233d02f5e3-tls-certs\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:41.078605 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.078577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899184eb-665d-4185-b920-29233d02f5e3-tls-certs\") pod \"model-serving-api-54d89bd5fc-p2cn6\" (UID: \"899184eb-665d-4185-b920-29233d02f5e3\") " pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:41.304968 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.304922 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" event={"ID":"8c767672-8156-4954-a189-dff457011f47","Type":"ContainerStarted","Data":"d71624a77ba203055d3305876db067875c67891faec274929a4c4b217ae94279"} Jul 29 07:19:41.306268 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.306240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" event={"ID":"56b3b447-9444-4e22-ab2a-f38dc1244cb0","Type":"ContainerStarted","Data":"d8d923f13c951516f3947318f35a3b98795cb8dee35324d2eb5005e6e6bca61b"} Jul 29 07:19:41.311499 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.311476 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:41.484058 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.484007 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p2cn6"] Jul 29 07:19:41.486880 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:19:41.486847 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899184eb_665d_4185_b920_29233d02f5e3.slice/crio-27794e4500c02b727c5dc9b99c7f0b2fcd81ee8068204d1d3f97657666e8b793 WatchSource:0}: Error finding container 27794e4500c02b727c5dc9b99c7f0b2fcd81ee8068204d1d3f97657666e8b793: Status 404 returned error can't find the container with id 27794e4500c02b727c5dc9b99c7f0b2fcd81ee8068204d1d3f97657666e8b793 Jul 29 07:19:41.591867 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:41.591707 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:42.315883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:42.315805 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" event={"ID":"899184eb-665d-4185-b920-29233d02f5e3","Type":"ContainerStarted","Data":"27794e4500c02b727c5dc9b99c7f0b2fcd81ee8068204d1d3f97657666e8b793"} Jul 29 07:19:45.593078 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:45.593023 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:19:45.593472 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:45.593108 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:19:46.337936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.337898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" event={"ID":"9e6a3c00-100f-4c4e-a5dd-1908529e2269","Type":"ContainerStarted","Data":"1234d3f4359eb8bae5fb5d47d6626424c9454c73fd66d6b8d61b3c95074ca4e5"} Jul 29 07:19:46.339706 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.339667 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" event={"ID":"8c767672-8156-4954-a189-dff457011f47","Type":"ContainerStarted","Data":"c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6"} Jul 29 07:19:46.339919 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.339818 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:46.341572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.341537 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" event={"ID":"56b3b447-9444-4e22-ab2a-f38dc1244cb0","Type":"ContainerStarted","Data":"63261cc2c608e4c42fe235ce9f3ef1e0a03d9bba2394ba50412223422637ae75"} Jul 29 07:19:46.341821 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.341697 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:46.342118 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.342095 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2g6cx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:19:46.342206 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.342156 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" podUID="8c767672-8156-4954-a189-dff457011f47" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:19:46.343524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.343503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" event={"ID":"899184eb-665d-4185-b920-29233d02f5e3","Type":"ContainerStarted","Data":"b7a51c2b30a42e501332a957c1228a7f80f929fefec3aa645888d74d6097eee4"} Jul 29 07:19:46.343726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.343708 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:19:46.366339 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.366274 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dd888fd86-hl6vg" podStartSLOduration=2.889891244 podStartE2EDuration="8.366251422s" totalImagesPullingTime="5.476360178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:46.363085853 +0000 UTC m=+296.134536652" watchObservedRunningTime="2026-07-29 07:19:46.366251422 +0000 UTC m=+296.137702223" Jul 29 07:19:46.393078 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.393007 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" podStartSLOduration=2.260475571 podStartE2EDuration="7.392989359s" totalImagesPullingTime="5.132513788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:46.391893715 +0000 UTC m=+296.163344515" watchObservedRunningTime="2026-07-29 07:19:46.392989359 +0000 UTC m=+296.164440175" Jul 29 07:19:46.416225 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.416165 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" podStartSLOduration=2.321034081 podStartE2EDuration="6.416148356s" totalImagesPullingTime="4.095114275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:46.415189359 +0000 UTC m=+296.186640159" watchObservedRunningTime="2026-07-29 07:19:46.416148356 +0000 UTC m=+296.187599154" Jul 29 07:19:46.450091 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:46.450014 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" podStartSLOduration=1.772427973 podStartE2EDuration="6.449996847s" totalImagesPullingTime="4.677568874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:46.448361685 +0000 UTC m=+296.219812476" watchObservedRunningTime="2026-07-29 07:19:46.449996847 +0000 UTC m=+296.221447645" Jul 29 07:19:47.237176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:47.237138 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:47.347844 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:47.347604 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:47.354453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:47.354418 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2g6cx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:19:47.354596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:47.354493 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" podUID="8c767672-8156-4954-a189-dff457011f47" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:19:48.149817 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:48.149782 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:48.395936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:48.395903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:49.669346 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.669301 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:49.804221 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.804183 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp"] Jul 29 07:19:49.807554 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.807532 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:49.815103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.815080 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 07:19:49.815103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.815094 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sbtkr\"" Jul 29 07:19:49.829425 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.829390 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:49.838419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.838387 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp"] Jul 29 07:19:49.913886 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.913849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:49.957530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.957424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6ba4b9-603b-430e-9d5c-023480140d23-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cl2zp\" (UID: \"9f6ba4b9-603b-430e-9d5c-023480140d23\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:49.957530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:49.957502 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkpt\" (UniqueName: \"kubernetes.io/projected/9f6ba4b9-603b-430e-9d5c-023480140d23-kube-api-access-trkpt\") pod \"llmisvc-controller-manager-777c6bd7d5-cl2zp\" (UID: \"9f6ba4b9-603b-430e-9d5c-023480140d23\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:50.011305 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.011266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:50.058211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.058174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkpt\" (UniqueName: \"kubernetes.io/projected/9f6ba4b9-603b-430e-9d5c-023480140d23-kube-api-access-trkpt\") pod \"llmisvc-controller-manager-777c6bd7d5-cl2zp\" (UID: \"9f6ba4b9-603b-430e-9d5c-023480140d23\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:50.058348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.058236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6ba4b9-603b-430e-9d5c-023480140d23-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cl2zp\" (UID: \"9f6ba4b9-603b-430e-9d5c-023480140d23\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:50.061069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.061026 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6ba4b9-603b-430e-9d5c-023480140d23-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cl2zp\" (UID: \"9f6ba4b9-603b-430e-9d5c-023480140d23\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:50.068099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.068072 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkpt\" (UniqueName: \"kubernetes.io/projected/9f6ba4b9-603b-430e-9d5c-023480140d23-kube-api-access-trkpt\") pod \"llmisvc-controller-manager-777c6bd7d5-cl2zp\" (UID: \"9f6ba4b9-603b-430e-9d5c-023480140d23\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:50.117603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.117569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:50.271855 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.271770 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp"] Jul 29 07:19:50.275403 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:19:50.275363 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f6ba4b9_603b_430e_9d5c_023480140d23.slice/crio-f3e1b67187d3763f1192c3c4bd73313d33ee7bac1cb33872cecc46700c887976 WatchSource:0}: Error finding container f3e1b67187d3763f1192c3c4bd73313d33ee7bac1cb33872cecc46700c887976: Status 404 returned error can't find the container with id f3e1b67187d3763f1192c3c4bd73313d33ee7bac1cb33872cecc46700c887976 Jul 29 07:19:50.354969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.354935 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:50.368660 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.368631 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" event={"ID":"9f6ba4b9-603b-430e-9d5c-023480140d23","Type":"ContainerStarted","Data":"f3e1b67187d3763f1192c3c4bd73313d33ee7bac1cb33872cecc46700c887976"} Jul 29 07:19:50.601296 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.601225 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:19:50.601469 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.601368 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:19:50.612467 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:50.612443 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 07:19:51.005572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.005542 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:51.107488 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.107457 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:51.207692 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.207652 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:51.322597 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.322517 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx"] Jul 29 07:19:51.322764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.322710 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" podUID="8c767672-8156-4954-a189-dff457011f47" containerName="discovery" containerID="cri-o://c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6" gracePeriod=30 Jul 29 07:19:51.743089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.743056 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:51.874944 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.874893 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c767672-8156-4954-a189-dff457011f47-local-certs\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.874959 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgc8k\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-kube-api-access-jgc8k\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875004 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767672-8156-4954-a189-dff457011f47-istio-csr-ca-configmap\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875067 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-csr-dns-cert\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875101 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-kubeconfig\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875146 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875148 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-istio-token\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875420 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875208 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-cacerts\") pod \"8c767672-8156-4954-a189-dff457011f47\" (UID: \"8c767672-8156-4954-a189-dff457011f47\") " Jul 29 07:19:51.875420 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875404 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c767672-8156-4954-a189-dff457011f47-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:19:51.875600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.875575 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767672-8156-4954-a189-dff457011f47-istio-csr-ca-configmap\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:51.878550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.878357 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:19:51.878550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.878477 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-kube-api-access-jgc8k" (OuterVolumeSpecName: "kube-api-access-jgc8k") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "kube-api-access-jgc8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:19:51.878755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.878658 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:19:51.878867 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.878839 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c767672-8156-4954-a189-dff457011f47-local-certs" (OuterVolumeSpecName: "local-certs") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:19:51.879277 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.879233 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-cacerts" (OuterVolumeSpecName: "cacerts") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:19:51.880143 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.880115 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8c767672-8156-4954-a189-dff457011f47" (UID: "8c767672-8156-4954-a189-dff457011f47"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:19:51.977142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.977024 2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c767672-8156-4954-a189-dff457011f47-local-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:51.977142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.977092 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgc8k\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-kube-api-access-jgc8k\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:51.977142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.977110 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-csr-dns-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:51.977142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.977124 2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-istio-kubeconfig\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:51.977142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.977138 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c767672-8156-4954-a189-dff457011f47-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:51.977672 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:51.977151 2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c767672-8156-4954-a189-dff457011f47-cacerts\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:19:52.379500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.379407 2568 generic.go:356] "Generic (PLEG): container finished" podID="8c767672-8156-4954-a189-dff457011f47" containerID="c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6" exitCode=0 Jul 29 07:19:52.379500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.379461 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" Jul 29 07:19:52.379500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.379476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" event={"ID":"8c767672-8156-4954-a189-dff457011f47","Type":"ContainerDied","Data":"c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6"} Jul 29 07:19:52.379977 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.379565 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx" event={"ID":"8c767672-8156-4954-a189-dff457011f47","Type":"ContainerDied","Data":"d71624a77ba203055d3305876db067875c67891faec274929a4c4b217ae94279"} Jul 29 07:19:52.379977 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.379588 2568 scope.go:117] "RemoveContainer" containerID="c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6" Jul 29 07:19:52.388832 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.388813 2568 scope.go:117] "RemoveContainer" containerID="c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6" Jul 29 07:19:52.389159 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:19:52.389138 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6\": container with ID starting with c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6 not found: ID does not exist" containerID="c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6" Jul 29 07:19:52.389233 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.389166 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6"} err="failed to get container status \"c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6\": rpc error: code = NotFound desc = could not find container \"c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6\": container with ID starting with c757e882f9f01ba692538014d46b9269dc78e7cc86f0d1e3beeae3c801e0a4f6 not found: ID does not exist" Jul 29 07:19:52.404367 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.404322 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx"] Jul 29 07:19:52.406402 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.406377 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2g6cx"] Jul 29 07:19:52.737765 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:52.737728 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c767672-8156-4954-a189-dff457011f47" path="/var/lib/kubelet/pods/8c767672-8156-4954-a189-dff457011f47/volumes" Jul 29 07:19:54.391485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:54.391448 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" event={"ID":"9f6ba4b9-603b-430e-9d5c-023480140d23","Type":"ContainerStarted","Data":"99e85c62c584447523fe7a21d2c69fcfa66bbe6f372c0137a08e9872c99d6d61"} Jul 29 07:19:54.391914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:54.391599 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:19:54.409730 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:54.409670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" podStartSLOduration=2.050404116 podStartE2EDuration="5.409651365s" totalImagesPullingTime="3.359247249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:54.408456328 +0000 UTC m=+304.179907129" watchObservedRunningTime="2026-07-29 07:19:54.409651365 +0000 UTC m=+304.181102163" Jul 29 07:19:57.356266 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:57.356236 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-dcg5q" Jul 29 07:19:57.358111 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:19:57.358090 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p2cn6" Jul 29 07:20:25.398131 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:25.398096 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cl2zp" Jul 29 07:20:39.998191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:39.998154 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:40.138643 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.138600 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfmw8"] Jul 29 07:20:40.139014 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.138996 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c767672-8156-4954-a189-dff457011f47" containerName="discovery" Jul 29 07:20:40.139144 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.139016 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c767672-8156-4954-a189-dff457011f47" containerName="discovery" Jul 29 07:20:40.139208 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.139148 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c767672-8156-4954-a189-dff457011f47" containerName="discovery" Jul 29 07:20:40.149082 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.148356 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:40.151956 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.151927 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 07:20:40.153179 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.153152 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-29wmp\"" Jul 29 07:20:40.153323 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.153300 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 07:20:40.154490 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.154216 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfmw8"] Jul 29 07:20:40.183100 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.183062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86vj\" (UniqueName: \"kubernetes.io/projected/e254885f-ae26-4b33-8acb-49696c7fed19-kube-api-access-b86vj\") pod \"kuadrant-operator-catalog-zfmw8\" (UID: \"e254885f-ae26-4b33-8acb-49696c7fed19\") " pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:40.284317 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.284222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86vj\" (UniqueName: \"kubernetes.io/projected/e254885f-ae26-4b33-8acb-49696c7fed19-kube-api-access-b86vj\") pod \"kuadrant-operator-catalog-zfmw8\" (UID: \"e254885f-ae26-4b33-8acb-49696c7fed19\") " pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:40.293840 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.293814 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86vj\" (UniqueName: \"kubernetes.io/projected/e254885f-ae26-4b33-8acb-49696c7fed19-kube-api-access-b86vj\") pod \"kuadrant-operator-catalog-zfmw8\" (UID: \"e254885f-ae26-4b33-8acb-49696c7fed19\") " pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:40.460175 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.460129 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:40.605337 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.605306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfmw8"] Jul 29 07:20:40.606774 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:20:40.606746 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode254885f_ae26_4b33_8acb_49696c7fed19.slice/crio-9c4e115257cfe4646ce0676f01c9bfb81536880dfafbfd811df20c3324ab603f WatchSource:0}: Error finding container 9c4e115257cfe4646ce0676f01c9bfb81536880dfafbfd811df20c3324ab603f: Status 404 returned error can't find the container with id 9c4e115257cfe4646ce0676f01c9bfb81536880dfafbfd811df20c3324ab603f Jul 29 07:20:40.608163 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.608144 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:20:40.661137 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.661102 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfmw8"] Jul 29 07:20:40.711286 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.711252 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5mxs6"] Jul 29 07:20:40.713726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.713708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:40.722919 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.722886 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5mxs6"] Jul 29 07:20:40.788630 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.788590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqg2\" (UniqueName: \"kubernetes.io/projected/f709d68f-78af-49a9-9ee0-b5b628b42de1-kube-api-access-2lqg2\") pod \"kuadrant-operator-catalog-5mxs6\" (UID: \"f709d68f-78af-49a9-9ee0-b5b628b42de1\") " pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:40.889742 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.889647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqg2\" (UniqueName: \"kubernetes.io/projected/f709d68f-78af-49a9-9ee0-b5b628b42de1-kube-api-access-2lqg2\") pod \"kuadrant-operator-catalog-5mxs6\" (UID: \"f709d68f-78af-49a9-9ee0-b5b628b42de1\") " pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:40.899573 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:40.899524 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqg2\" (UniqueName: \"kubernetes.io/projected/f709d68f-78af-49a9-9ee0-b5b628b42de1-kube-api-access-2lqg2\") pod \"kuadrant-operator-catalog-5mxs6\" (UID: \"f709d68f-78af-49a9-9ee0-b5b628b42de1\") " pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:41.024199 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:41.024153 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:41.178558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:41.178517 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5mxs6"] Jul 29 07:20:41.181534 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:20:41.181504 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf709d68f_78af_49a9_9ee0_b5b628b42de1.slice/crio-9d5675146cac2b81e52185ef9d198db9d5dd0eb66a66a97de16062a4d71d6c4b WatchSource:0}: Error finding container 9d5675146cac2b81e52185ef9d198db9d5dd0eb66a66a97de16062a4d71d6c4b: Status 404 returned error can't find the container with id 9d5675146cac2b81e52185ef9d198db9d5dd0eb66a66a97de16062a4d71d6c4b Jul 29 07:20:41.426410 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:41.426329 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:41.610942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:41.610843 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" event={"ID":"f709d68f-78af-49a9-9ee0-b5b628b42de1","Type":"ContainerStarted","Data":"9d5675146cac2b81e52185ef9d198db9d5dd0eb66a66a97de16062a4d71d6c4b"} Jul 29 07:20:41.612237 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:41.612205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" event={"ID":"e254885f-ae26-4b33-8acb-49696c7fed19","Type":"ContainerStarted","Data":"9c4e115257cfe4646ce0676f01c9bfb81536880dfafbfd811df20c3324ab603f"} Jul 29 07:20:43.623715 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.623677 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" event={"ID":"f709d68f-78af-49a9-9ee0-b5b628b42de1","Type":"ContainerStarted","Data":"f9b202d34ddc73403958f44d22965775d4d89a4c2f3d377ea0065431d2858b0f"} Jul 29 07:20:43.625092 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.625070 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" event={"ID":"e254885f-ae26-4b33-8acb-49696c7fed19","Type":"ContainerStarted","Data":"2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939"} Jul 29 07:20:43.625168 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.625114 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" podUID="e254885f-ae26-4b33-8acb-49696c7fed19" containerName="registry-server" containerID="cri-o://2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939" gracePeriod=2 Jul 29 07:20:43.641264 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.641220 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" podStartSLOduration=2.019090587 podStartE2EDuration="3.641196312s" totalImagesPullingTime="1.622105725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:20:43.640938164 +0000 UTC m=+353.412388964" watchObservedRunningTime="2026-07-29 07:20:43.641196312 +0000 UTC m=+353.412647111" Jul 29 07:20:43.656134 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.656083 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" podStartSLOduration=1.455381937 podStartE2EDuration="3.656067015s" totalImagesPullingTime="2.200685078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:20:43.654633702 +0000 UTC m=+353.426084501" watchObservedRunningTime="2026-07-29 07:20:43.656067015 +0000 UTC m=+353.427517811" Jul 29 07:20:43.866033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.866001 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:43.916990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.916958 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b86vj\" (UniqueName: \"kubernetes.io/projected/e254885f-ae26-4b33-8acb-49696c7fed19-kube-api-access-b86vj\") pod \"e254885f-ae26-4b33-8acb-49696c7fed19\" (UID: \"e254885f-ae26-4b33-8acb-49696c7fed19\") " Jul 29 07:20:43.919372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:43.919336 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e254885f-ae26-4b33-8acb-49696c7fed19-kube-api-access-b86vj" (OuterVolumeSpecName: "kube-api-access-b86vj") pod "e254885f-ae26-4b33-8acb-49696c7fed19" (UID: "e254885f-ae26-4b33-8acb-49696c7fed19"). InnerVolumeSpecName "kube-api-access-b86vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:20:44.017684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.017646 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86vj\" (UniqueName: \"kubernetes.io/projected/e254885f-ae26-4b33-8acb-49696c7fed19-kube-api-access-b86vj\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:20:44.634186 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.634155 2568 generic.go:356] "Generic (PLEG): container finished" podID="e254885f-ae26-4b33-8acb-49696c7fed19" containerID="2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939" exitCode=0 Jul 29 07:20:44.634664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.634203 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" Jul 29 07:20:44.634664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.634253 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" event={"ID":"e254885f-ae26-4b33-8acb-49696c7fed19","Type":"ContainerDied","Data":"2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939"} Jul 29 07:20:44.634664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.634296 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zfmw8" event={"ID":"e254885f-ae26-4b33-8acb-49696c7fed19","Type":"ContainerDied","Data":"9c4e115257cfe4646ce0676f01c9bfb81536880dfafbfd811df20c3324ab603f"} Jul 29 07:20:44.634664 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.634321 2568 scope.go:117] "RemoveContainer" containerID="2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939" Jul 29 07:20:44.644305 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.644285 2568 scope.go:117] "RemoveContainer" containerID="2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939" Jul 29 07:20:44.644609 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:20:44.644589 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939\": container with ID starting with 2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939 not found: ID does not exist" containerID="2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939" Jul 29 07:20:44.644670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.644621 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939"} err="failed to get container status \"2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939\": rpc error: code = NotFound desc = could not find container \"2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939\": container with ID starting with 2f4605d3bb40f6db6e894cb08ed1f78f5b3ae7136bde01ff598b802d73c60939 not found: ID does not exist" Jul 29 07:20:44.659579 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.659538 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfmw8"] Jul 29 07:20:44.661577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.661552 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfmw8"] Jul 29 07:20:44.738212 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:44.738175 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e254885f-ae26-4b33-8acb-49696c7fed19" path="/var/lib/kubelet/pods/e254885f-ae26-4b33-8acb-49696c7fed19/volumes" Jul 29 07:20:51.024740 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:51.024696 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:51.025194 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:51.024764 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:51.047455 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:51.047424 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:20:51.689923 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:20:51.689886 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-5mxs6" Jul 29 07:21:08.312742 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:08.312702 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:11.338742 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.338698 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b"] Jul 29 07:21:11.339263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.339244 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e254885f-ae26-4b33-8acb-49696c7fed19" containerName="registry-server" Jul 29 07:21:11.339334 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.339266 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e254885f-ae26-4b33-8acb-49696c7fed19" containerName="registry-server" Jul 29 07:21:11.339388 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.339377 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e254885f-ae26-4b33-8acb-49696c7fed19" containerName="registry-server" Jul 29 07:21:11.342333 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.342308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:11.345725 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.345705 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qvx8x\"" Jul 29 07:21:11.364308 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.364275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b"] Jul 29 07:21:11.465591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.465553 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwjh\" (UniqueName: \"kubernetes.io/projected/4e954a47-d1a9-4692-ad07-83789a137208-kube-api-access-slwjh\") pod \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" (UID: \"4e954a47-d1a9-4692-ad07-83789a137208\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:11.566314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.566271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwjh\" (UniqueName: \"kubernetes.io/projected/4e954a47-d1a9-4692-ad07-83789a137208-kube-api-access-slwjh\") pod \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" (UID: \"4e954a47-d1a9-4692-ad07-83789a137208\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:11.582976 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.582936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwjh\" (UniqueName: \"kubernetes.io/projected/4e954a47-d1a9-4692-ad07-83789a137208-kube-api-access-slwjh\") pod \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" (UID: \"4e954a47-d1a9-4692-ad07-83789a137208\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:11.653066 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.653000 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:11.822641 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:11.822615 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b"] Jul 29 07:21:11.825309 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:21:11.825273 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e954a47_d1a9_4692_ad07_83789a137208.slice/crio-c0dadd0bfe8346a844bef2e7f276ad57e519639b37b112e91b54e701fd37cfd4 WatchSource:0}: Error finding container c0dadd0bfe8346a844bef2e7f276ad57e519639b37b112e91b54e701fd37cfd4: Status 404 returned error can't find the container with id c0dadd0bfe8346a844bef2e7f276ad57e519639b37b112e91b54e701fd37cfd4 Jul 29 07:21:12.094910 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:12.094814 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:12.785861 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:12.785824 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" event={"ID":"4e954a47-d1a9-4692-ad07-83789a137208","Type":"ContainerStarted","Data":"c0dadd0bfe8346a844bef2e7f276ad57e519639b37b112e91b54e701fd37cfd4"} Jul 29 07:21:13.220914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:13.220871 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:13.792316 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:13.792219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" event={"ID":"4e954a47-d1a9-4692-ad07-83789a137208","Type":"ContainerStarted","Data":"5694e5fb56fbee8ff4ddb4959a379c2987488595fe520ad54126d08cade01ba2"} Jul 29 07:21:13.792316 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:13.792283 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:13.810637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:13.810583 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" podStartSLOduration=1.242483059 podStartE2EDuration="2.810566428s" totalImagesPullingTime="1.568083369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:13.809386403 +0000 UTC m=+383.580837203" watchObservedRunningTime="2026-07-29 07:21:13.810566428 +0000 UTC m=+383.582017226" Jul 29 07:21:16.150773 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.150732 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm"] Jul 29 07:21:16.154364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.154343 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:16.159189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.159166 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z4d8k\"" Jul 29 07:21:16.163701 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.163674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm"] Jul 29 07:21:16.214429 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.214391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxj2\" (UniqueName: \"kubernetes.io/projected/e954145a-c0c8-403f-b4f5-ffeafd0464c1-kube-api-access-tlxj2\") pod \"authorino-operator-75cc8d5cb7-hdkjm\" (UID: \"e954145a-c0c8-403f-b4f5-ffeafd0464c1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:16.315796 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.315762 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxj2\" (UniqueName: \"kubernetes.io/projected/e954145a-c0c8-403f-b4f5-ffeafd0464c1-kube-api-access-tlxj2\") pod \"authorino-operator-75cc8d5cb7-hdkjm\" (UID: \"e954145a-c0c8-403f-b4f5-ffeafd0464c1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:16.328216 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.328182 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxj2\" (UniqueName: \"kubernetes.io/projected/e954145a-c0c8-403f-b4f5-ffeafd0464c1-kube-api-access-tlxj2\") pod \"authorino-operator-75cc8d5cb7-hdkjm\" (UID: \"e954145a-c0c8-403f-b4f5-ffeafd0464c1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:16.465383 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.465320 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:16.617973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.617917 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm"] Jul 29 07:21:16.620312 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:21:16.620274 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode954145a_c0c8_403f_b4f5_ffeafd0464c1.slice/crio-f5c39dc1161687f2703190bb8eb4e8be7ba78d0bc209b5b9fc7e2831ea42b725 WatchSource:0}: Error finding container f5c39dc1161687f2703190bb8eb4e8be7ba78d0bc209b5b9fc7e2831ea42b725: Status 404 returned error can't find the container with id f5c39dc1161687f2703190bb8eb4e8be7ba78d0bc209b5b9fc7e2831ea42b725 Jul 29 07:21:16.815956 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:16.815866 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" event={"ID":"e954145a-c0c8-403f-b4f5-ffeafd0464c1","Type":"ContainerStarted","Data":"f5c39dc1161687f2703190bb8eb4e8be7ba78d0bc209b5b9fc7e2831ea42b725"} Jul 29 07:21:17.719600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:17.719523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:19.839820 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:19.839780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" event={"ID":"e954145a-c0c8-403f-b4f5-ffeafd0464c1","Type":"ContainerStarted","Data":"cd5d12108f8947a48aae31aec3cb53143adfccf0b6ec7eb7aa654f02928c5700"} Jul 29 07:21:19.840285 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:19.839886 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:19.856508 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:19.856453 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" podStartSLOduration=1.512818182 podStartE2EDuration="3.856436786s" totalImagesPullingTime="2.343618604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:19.855918338 +0000 UTC m=+389.627369150" watchObservedRunningTime="2026-07-29 07:21:19.856436786 +0000 UTC m=+389.627887583" Jul 29 07:21:20.325081 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.325030 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc"] Jul 29 07:21:20.328529 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.328507 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:20.331859 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.331834 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zzpvt\"" Jul 29 07:21:20.331979 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.331834 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 07:21:20.344196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.344156 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc"] Jul 29 07:21:20.352133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.352098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whl25\" (UniqueName: \"kubernetes.io/projected/c286b451-056f-44b7-bf23-c5b26e3997f9-kube-api-access-whl25\") pod \"dns-operator-controller-manager-864dd78c4c-b5gtc\" (UID: \"c286b451-056f-44b7-bf23-c5b26e3997f9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:20.452815 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.452774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whl25\" (UniqueName: \"kubernetes.io/projected/c286b451-056f-44b7-bf23-c5b26e3997f9-kube-api-access-whl25\") pod \"dns-operator-controller-manager-864dd78c4c-b5gtc\" (UID: \"c286b451-056f-44b7-bf23-c5b26e3997f9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:20.464407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.464378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whl25\" (UniqueName: \"kubernetes.io/projected/c286b451-056f-44b7-bf23-c5b26e3997f9-kube-api-access-whl25\") pod \"dns-operator-controller-manager-864dd78c4c-b5gtc\" (UID: \"c286b451-056f-44b7-bf23-c5b26e3997f9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:20.640421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.640314 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:20.749271 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.749198 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:20.819794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.819770 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc"] Jul 29 07:21:20.822243 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:21:20.822215 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc286b451_056f_44b7_bf23_c5b26e3997f9.slice/crio-442c9b55962088e02b324e2ecee87b49e1014fda842bab3b8f09b8478367de9a WatchSource:0}: Error finding container 442c9b55962088e02b324e2ecee87b49e1014fda842bab3b8f09b8478367de9a: Status 404 returned error can't find the container with id 442c9b55962088e02b324e2ecee87b49e1014fda842bab3b8f09b8478367de9a Jul 29 07:21:20.846023 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.845991 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" event={"ID":"c286b451-056f-44b7-bf23-c5b26e3997f9","Type":"ContainerStarted","Data":"442c9b55962088e02b324e2ecee87b49e1014fda842bab3b8f09b8478367de9a"} Jul 29 07:21:20.958740 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:20.958698 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:22.859646 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:22.859609 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" event={"ID":"c286b451-056f-44b7-bf23-c5b26e3997f9","Type":"ContainerStarted","Data":"03482026dc1b7350665088e1cf70c6c9baeacff018009512067800a1579250c4"} Jul 29 07:21:22.860068 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:22.859743 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:22.877407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:22.877323 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" podStartSLOduration=1.067379864 podStartE2EDuration="2.877304598s" totalImagesPullingTime="1.809924734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:22.876659573 +0000 UTC m=+392.648110372" watchObservedRunningTime="2026-07-29 07:21:22.877304598 +0000 UTC m=+392.648755481" Jul 29 07:21:23.140166 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:23.140129 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:24.740856 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:24.740795 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:24.807685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:24.807647 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:26.401126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.401083 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k"] Jul 29 07:21:26.404982 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.404961 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.408372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.408343 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nbw6s\"" Jul 29 07:21:26.420467 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.420432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k"] Jul 29 07:21:26.504924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.504889 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg7v\" (UniqueName: \"kubernetes.io/projected/75f7a068-c68f-4961-896a-ee48dcc3f605-kube-api-access-2wg7v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.505153 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.505025 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f7a068-c68f-4961-896a-ee48dcc3f605-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.606183 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.606134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f7a068-c68f-4961-896a-ee48dcc3f605-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.606369 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.606207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg7v\" (UniqueName: \"kubernetes.io/projected/75f7a068-c68f-4961-896a-ee48dcc3f605-kube-api-access-2wg7v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.606661 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.606639 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f7a068-c68f-4961-896a-ee48dcc3f605-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.623702 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.623666 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg7v\" (UniqueName: \"kubernetes.io/projected/75f7a068-c68f-4961-896a-ee48dcc3f605-kube-api-access-2wg7v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.716206 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.716168 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:26.869403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.869378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k"] Jul 29 07:21:26.871841 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:21:26.871806 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f7a068_c68f_4961_896a_ee48dcc3f605.slice/crio-a2ef63ac0f8ba2b1f29d082dfc43afdc240c80fc536b895a90f3e5158e6ec1f9 WatchSource:0}: Error finding container a2ef63ac0f8ba2b1f29d082dfc43afdc240c80fc536b895a90f3e5158e6ec1f9: Status 404 returned error can't find the container with id a2ef63ac0f8ba2b1f29d082dfc43afdc240c80fc536b895a90f3e5158e6ec1f9 Jul 29 07:21:26.882231 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:26.882194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" event={"ID":"75f7a068-c68f-4961-896a-ee48dcc3f605","Type":"ContainerStarted","Data":"a2ef63ac0f8ba2b1f29d082dfc43afdc240c80fc536b895a90f3e5158e6ec1f9"} Jul 29 07:21:27.134237 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:27.134140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:29.296115 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:29.296064 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:29.397800 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:29.397712 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:30.848833 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:30.848800 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hdkjm" Jul 29 07:21:32.922658 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:32.922556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" event={"ID":"75f7a068-c68f-4961-896a-ee48dcc3f605","Type":"ContainerStarted","Data":"408a6825cf5c0f9ea69b57e34f98acbc17ce4c00fdac3c6e906411bb808e1678"} Jul 29 07:21:32.923292 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:32.922790 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:32.947364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:32.947304 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" podStartSLOduration=1.8945841350000001 podStartE2EDuration="6.947286855s" totalImagesPullingTime="5.05270272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:32.944650943 +0000 UTC m=+402.716101742" watchObservedRunningTime="2026-07-29 07:21:32.947286855 +0000 UTC m=+402.718737715" Jul 29 07:21:33.867650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:33.867613 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5gtc" Jul 29 07:21:34.331381 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:34.331340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:34.466592 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:34.466548 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:34.854901 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:34.854862 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:34.950035 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:34.949997 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:35.049148 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:35.049114 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:35.884595 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:35.884558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:35.980067 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:35.980001 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:36.080753 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:36.080699 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:36.611505 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:36.611460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:36.687542 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:36.687502 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:39.300546 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:39.300511 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:39.411885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:39.411843 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:43.101203 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:43.101165 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:43.159006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:43.158969 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:43.931660 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:43.931623 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:45.780125 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.780083 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k"] Jul 29 07:21:45.780595 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.780279 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" containerName="manager" containerID="cri-o://408a6825cf5c0f9ea69b57e34f98acbc17ce4c00fdac3c6e906411bb808e1678" gracePeriod=2 Jul 29 07:21:45.791175 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.791144 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k"] Jul 29 07:21:45.794924 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.794878 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75f7a068-c68f-4961-896a-ee48dcc3f605\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-b659k\": pods \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:45.805715 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.805682 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7"] Jul 29 07:21:45.806167 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.806142 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" containerName="manager" Jul 29 07:21:45.806167 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.806163 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" containerName="manager" Jul 29 07:21:45.806337 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.806253 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" containerName="manager" Jul 29 07:21:45.809600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.809559 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b"] Jul 29 07:21:45.809756 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.809696 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:45.809885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.809775 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" podUID="4e954a47-d1a9-4692-ad07-83789a137208" containerName="manager" containerID="cri-o://5694e5fb56fbee8ff4ddb4959a379c2987488595fe520ad54126d08cade01ba2" gracePeriod=2 Jul 29 07:21:45.818998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.818960 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b"] Jul 29 07:21:45.822949 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.822895 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7"] Jul 29 07:21:45.824896 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.824846 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4e954a47-d1a9-4692-ad07-83789a137208\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-xkg4b\": pods \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:45.827292 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.827249 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:45.829920 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.829887 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:45.832274 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.832234 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="4e954a47-d1a9-4692-ad07-83789a137208" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:45.833028 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.833004 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2"] Jul 29 07:21:45.833616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.833596 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e954a47-d1a9-4692-ad07-83789a137208" containerName="manager" Jul 29 07:21:45.833680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.833623 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e954a47-d1a9-4692-ad07-83789a137208" containerName="manager" Jul 29 07:21:45.833739 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.833729 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e954a47-d1a9-4692-ad07-83789a137208" containerName="manager" Jul 29 07:21:45.836881 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.836857 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:45.853763 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.853723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2"] Jul 29 07:21:45.858128 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.858086 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="4e954a47-d1a9-4692-ad07-83789a137208" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:45.880371 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:45.880329 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:45.991516 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.991292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6beef9ba-d06c-486c-8187-b5ee550a4ffb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wblm7\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:45.991516 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.991350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2vj\" (UniqueName: \"kubernetes.io/projected/6beef9ba-d06c-486c-8187-b5ee550a4ffb-kube-api-access-tj2vj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wblm7\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:45.991516 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.991395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ng28\" (UniqueName: \"kubernetes.io/projected/f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8-kube-api-access-9ng28\") pod \"limitador-operator-controller-manager-859d7b78bd-n69t2\" (UID: \"f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:45.998530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:45.998497 2568 generic.go:356] "Generic (PLEG): container finished" podID="4e954a47-d1a9-4692-ad07-83789a137208" containerID="5694e5fb56fbee8ff4ddb4959a379c2987488595fe520ad54126d08cade01ba2" exitCode=0 Jul 29 07:21:46.000669 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.000642 2568 generic.go:356] "Generic (PLEG): container finished" podID="75f7a068-c68f-4961-896a-ee48dcc3f605" containerID="408a6825cf5c0f9ea69b57e34f98acbc17ce4c00fdac3c6e906411bb808e1678" exitCode=0 Jul 29 07:21:46.071870 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.071844 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:46.074420 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:46.074392 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="4e954a47-d1a9-4692-ad07-83789a137208" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:46.075190 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.075173 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:46.076870 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:46.076841 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:46.080915 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:46.080883 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xkg4b\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="4e954a47-d1a9-4692-ad07-83789a137208" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:46.083641 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:21:46.083615 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-b659k\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:46.092829 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.092791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6beef9ba-d06c-486c-8187-b5ee550a4ffb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wblm7\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:46.093009 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.092840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2vj\" (UniqueName: \"kubernetes.io/projected/6beef9ba-d06c-486c-8187-b5ee550a4ffb-kube-api-access-tj2vj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wblm7\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:46.093009 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.092872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ng28\" (UniqueName: \"kubernetes.io/projected/f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8-kube-api-access-9ng28\") pod \"limitador-operator-controller-manager-859d7b78bd-n69t2\" (UID: \"f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:46.093231 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.093208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6beef9ba-d06c-486c-8187-b5ee550a4ffb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wblm7\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:46.106794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.106762 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ng28\" (UniqueName: \"kubernetes.io/projected/f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8-kube-api-access-9ng28\") pod \"limitador-operator-controller-manager-859d7b78bd-n69t2\" (UID: \"f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:46.106794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.106766 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2vj\" (UniqueName: \"kubernetes.io/projected/6beef9ba-d06c-486c-8187-b5ee550a4ffb-kube-api-access-tj2vj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wblm7\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:46.193627 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.193589 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wg7v\" (UniqueName: \"kubernetes.io/projected/75f7a068-c68f-4961-896a-ee48dcc3f605-kube-api-access-2wg7v\") pod \"75f7a068-c68f-4961-896a-ee48dcc3f605\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " Jul 29 07:21:46.193828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.193660 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slwjh\" (UniqueName: \"kubernetes.io/projected/4e954a47-d1a9-4692-ad07-83789a137208-kube-api-access-slwjh\") pod \"4e954a47-d1a9-4692-ad07-83789a137208\" (UID: \"4e954a47-d1a9-4692-ad07-83789a137208\") " Jul 29 07:21:46.193828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.193743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f7a068-c68f-4961-896a-ee48dcc3f605-extensions-socket-volume\") pod \"75f7a068-c68f-4961-896a-ee48dcc3f605\" (UID: \"75f7a068-c68f-4961-896a-ee48dcc3f605\") " Jul 29 07:21:46.194274 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.194243 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f7a068-c68f-4961-896a-ee48dcc3f605-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "75f7a068-c68f-4961-896a-ee48dcc3f605" (UID: "75f7a068-c68f-4961-896a-ee48dcc3f605"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:46.196005 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.195986 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e954a47-d1a9-4692-ad07-83789a137208-kube-api-access-slwjh" (OuterVolumeSpecName: "kube-api-access-slwjh") pod "4e954a47-d1a9-4692-ad07-83789a137208" (UID: "4e954a47-d1a9-4692-ad07-83789a137208"). InnerVolumeSpecName "kube-api-access-slwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:46.196084 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.195994 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f7a068-c68f-4961-896a-ee48dcc3f605-kube-api-access-2wg7v" (OuterVolumeSpecName: "kube-api-access-2wg7v") pod "75f7a068-c68f-4961-896a-ee48dcc3f605" (UID: "75f7a068-c68f-4961-896a-ee48dcc3f605"). InnerVolumeSpecName "kube-api-access-2wg7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:46.227974 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.227935 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:46.236206 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.236167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:46.297353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.297214 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwjh\" (UniqueName: \"kubernetes.io/projected/4e954a47-d1a9-4692-ad07-83789a137208-kube-api-access-slwjh\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:21:46.297353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.297258 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f7a068-c68f-4961-896a-ee48dcc3f605-extensions-socket-volume\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:21:46.297353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.297272 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wg7v\" (UniqueName: \"kubernetes.io/projected/75f7a068-c68f-4961-896a-ee48dcc3f605-kube-api-access-2wg7v\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:21:46.394612 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.394586 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7"] Jul 29 07:21:46.396675 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:21:46.396651 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6beef9ba_d06c_486c_8187_b5ee550a4ffb.slice/crio-e0080dbbffd4255fd6167d932afb697d58868591e4dd2c8c805a76150eaa3093 WatchSource:0}: Error finding container e0080dbbffd4255fd6167d932afb697d58868591e4dd2c8c805a76150eaa3093: Status 404 returned error can't find the container with id e0080dbbffd4255fd6167d932afb697d58868591e4dd2c8c805a76150eaa3093 Jul 29 07:21:46.423620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.423584 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2"] Jul 29 07:21:46.433383 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:21:46.433352 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d393ce_18fe_4870_9b95_2a2e8bdaf8c8.slice/crio-bc8e58c2b7949de64f25d0d53412f9b2543168ba3f08e66431cfaf07c2581383 WatchSource:0}: Error finding container bc8e58c2b7949de64f25d0d53412f9b2543168ba3f08e66431cfaf07c2581383: Status 404 returned error can't find the container with id bc8e58c2b7949de64f25d0d53412f9b2543168ba3f08e66431cfaf07c2581383 Jul 29 07:21:46.738354 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.738318 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e954a47-d1a9-4692-ad07-83789a137208" path="/var/lib/kubelet/pods/4e954a47-d1a9-4692-ad07-83789a137208/volumes" Jul 29 07:21:46.738680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:46.738666 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f7a068-c68f-4961-896a-ee48dcc3f605" path="/var/lib/kubelet/pods/75f7a068-c68f-4961-896a-ee48dcc3f605/volumes" Jul 29 07:21:47.009893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.009799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" event={"ID":"f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8","Type":"ContainerStarted","Data":"e78b49eaf9adf7c9307deb2d2ea213a3786667f66e62af15876f1992093231fe"} Jul 29 07:21:47.009893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.009858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" event={"ID":"f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8","Type":"ContainerStarted","Data":"bc8e58c2b7949de64f25d0d53412f9b2543168ba3f08e66431cfaf07c2581383"} Jul 29 07:21:47.010416 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.009896 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:47.011508 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.011478 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xkg4b" Jul 29 07:21:47.011657 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.011483 2568 scope.go:117] "RemoveContainer" containerID="5694e5fb56fbee8ff4ddb4959a379c2987488595fe520ad54126d08cade01ba2" Jul 29 07:21:47.013566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.013529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" event={"ID":"6beef9ba-d06c-486c-8187-b5ee550a4ffb","Type":"ContainerStarted","Data":"68370efe4928c750015cfb7fda46a3bd62a8bff10ca7a1b72e8f924663ed10cb"} Jul 29 07:21:47.013685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.013594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" event={"ID":"6beef9ba-d06c-486c-8187-b5ee550a4ffb","Type":"ContainerStarted","Data":"e0080dbbffd4255fd6167d932afb697d58868591e4dd2c8c805a76150eaa3093"} Jul 29 07:21:47.013685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.013626 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:21:47.015602 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.015581 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b659k" Jul 29 07:21:47.023121 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.023096 2568 scope.go:117] "RemoveContainer" containerID="408a6825cf5c0f9ea69b57e34f98acbc17ce4c00fdac3c6e906411bb808e1678" Jul 29 07:21:47.046852 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.046789 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" podStartSLOduration=2.046768636 podStartE2EDuration="2.046768636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:47.045430765 +0000 UTC m=+416.816881566" watchObservedRunningTime="2026-07-29 07:21:47.046768636 +0000 UTC m=+416.818219435" Jul 29 07:21:47.061572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.061532 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:47.072993 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.072945 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" podStartSLOduration=2.072928356 podStartE2EDuration="2.072928356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:47.071909876 +0000 UTC m=+416.843360676" watchObservedRunningTime="2026-07-29 07:21:47.072928356 +0000 UTC m=+416.844379155" Jul 29 07:21:47.231873 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.231835 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:47.331729 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:47.331638 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:49.482262 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:49.482224 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:49.583183 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:49.583147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:52.859294 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:52.859252 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:52.958628 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:52.958585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:55.298762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:55.298728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:55.395386 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:55.395349 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:58.025577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:58.025542 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n69t2" Jul 29 07:21:58.025989 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:21:58.025616 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:22:00.624813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:00.624772 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:00.726902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:00.726861 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:01.924330 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:01.924294 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7"] Jul 29 07:22:01.924822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:01.924515 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" podUID="6beef9ba-d06c-486c-8187-b5ee550a4ffb" containerName="manager" containerID="cri-o://68370efe4928c750015cfb7fda46a3bd62a8bff10ca7a1b72e8f924663ed10cb" gracePeriod=10 Jul 29 07:22:02.097139 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.097106 2568 generic.go:356] "Generic (PLEG): container finished" podID="6beef9ba-d06c-486c-8187-b5ee550a4ffb" containerID="68370efe4928c750015cfb7fda46a3bd62a8bff10ca7a1b72e8f924663ed10cb" exitCode=0 Jul 29 07:22:02.097304 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.097176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" event={"ID":"6beef9ba-d06c-486c-8187-b5ee550a4ffb","Type":"ContainerDied","Data":"68370efe4928c750015cfb7fda46a3bd62a8bff10ca7a1b72e8f924663ed10cb"} Jul 29 07:22:02.175885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.175856 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27"] Jul 29 07:22:02.178290 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.178270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.190578 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.190552 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:22:02.204496 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.204131 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27"] Jul 29 07:22:02.216427 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.216393 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj2vj\" (UniqueName: \"kubernetes.io/projected/6beef9ba-d06c-486c-8187-b5ee550a4ffb-kube-api-access-tj2vj\") pod \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " Jul 29 07:22:02.216596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.216543 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8071ae2d-f735-4489-b7c1-3f3080b9db36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bbm27\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.216596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.216567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rzq\" (UniqueName: \"kubernetes.io/projected/8071ae2d-f735-4489-b7c1-3f3080b9db36-kube-api-access-s6rzq\") pod \"kuadrant-operator-controller-manager-86878c769f-bbm27\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.219142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.219104 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6beef9ba-d06c-486c-8187-b5ee550a4ffb-kube-api-access-tj2vj" (OuterVolumeSpecName: "kube-api-access-tj2vj") pod "6beef9ba-d06c-486c-8187-b5ee550a4ffb" (UID: "6beef9ba-d06c-486c-8187-b5ee550a4ffb"). InnerVolumeSpecName "kube-api-access-tj2vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:02.316865 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.316828 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6beef9ba-d06c-486c-8187-b5ee550a4ffb-extensions-socket-volume\") pod \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\" (UID: \"6beef9ba-d06c-486c-8187-b5ee550a4ffb\") " Jul 29 07:22:02.317072 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.316964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8071ae2d-f735-4489-b7c1-3f3080b9db36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bbm27\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.317072 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.316985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rzq\" (UniqueName: \"kubernetes.io/projected/8071ae2d-f735-4489-b7c1-3f3080b9db36-kube-api-access-s6rzq\") pod \"kuadrant-operator-controller-manager-86878c769f-bbm27\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.317072 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.317012 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj2vj\" (UniqueName: \"kubernetes.io/projected/6beef9ba-d06c-486c-8187-b5ee550a4ffb-kube-api-access-tj2vj\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:22:02.317251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.317226 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6beef9ba-d06c-486c-8187-b5ee550a4ffb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6beef9ba-d06c-486c-8187-b5ee550a4ffb" (UID: "6beef9ba-d06c-486c-8187-b5ee550a4ffb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:02.317373 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.317353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8071ae2d-f735-4489-b7c1-3f3080b9db36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bbm27\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.326650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.326618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rzq\" (UniqueName: \"kubernetes.io/projected/8071ae2d-f735-4489-b7c1-3f3080b9db36-kube-api-access-s6rzq\") pod \"kuadrant-operator-controller-manager-86878c769f-bbm27\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.418123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.418090 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6beef9ba-d06c-486c-8187-b5ee550a4ffb-extensions-socket-volume\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:22:02.502539 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.502441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:02.650673 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:02.650604 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27"] Jul 29 07:22:02.653021 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:22:02.652985 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8071ae2d_f735_4489_b7c1_3f3080b9db36.slice/crio-bd059bacc3b5beaddbdf6b573fbfa7e70615bb573467569526c0e7da58558450 WatchSource:0}: Error finding container bd059bacc3b5beaddbdf6b573fbfa7e70615bb573467569526c0e7da58558450: Status 404 returned error can't find the container with id bd059bacc3b5beaddbdf6b573fbfa7e70615bb573467569526c0e7da58558450 Jul 29 07:22:03.104122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.104091 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" Jul 29 07:22:03.104122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.104096 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7" event={"ID":"6beef9ba-d06c-486c-8187-b5ee550a4ffb","Type":"ContainerDied","Data":"e0080dbbffd4255fd6167d932afb697d58868591e4dd2c8c805a76150eaa3093"} Jul 29 07:22:03.104681 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.104166 2568 scope.go:117] "RemoveContainer" containerID="68370efe4928c750015cfb7fda46a3bd62a8bff10ca7a1b72e8f924663ed10cb" Jul 29 07:22:03.106370 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.106318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" event={"ID":"8071ae2d-f735-4489-b7c1-3f3080b9db36","Type":"ContainerStarted","Data":"95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd"} Jul 29 07:22:03.106474 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.106377 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" event={"ID":"8071ae2d-f735-4489-b7c1-3f3080b9db36","Type":"ContainerStarted","Data":"bd059bacc3b5beaddbdf6b573fbfa7e70615bb573467569526c0e7da58558450"} Jul 29 07:22:03.106547 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.106498 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:03.132172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.132116 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" podStartSLOduration=1.132101029 podStartE2EDuration="1.132101029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:22:03.131636353 +0000 UTC m=+432.903087147" watchObservedRunningTime="2026-07-29 07:22:03.132101029 +0000 UTC m=+432.903551828" Jul 29 07:22:03.154398 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.154343 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7"] Jul 29 07:22:03.159129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:03.159084 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wblm7"] Jul 29 07:22:04.738030 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:04.737981 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6beef9ba-d06c-486c-8187-b5ee550a4ffb" path="/var/lib/kubelet/pods/6beef9ba-d06c-486c-8187-b5ee550a4ffb/volumes" Jul 29 07:22:05.776788 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:05.776752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:07.315689 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:07.315656 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:07.418059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:07.417991 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:11.040719 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:11.040678 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:11.141771 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:11.141729 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:11.265572 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:11.265536 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:11.414424 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:11.414390 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:14.115933 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:14.115898 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:22:15.134949 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:15.134908 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:15.220364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:15.220325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:18.407267 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:18.407227 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:18.885510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:18.885418 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:20.453384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:20.453350 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:20.554299 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:20.554251 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:29.470220 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:29.470182 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:29.538401 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:29.538365 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:34.932092 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.932022 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:34.943837 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.943796 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:34.944239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.944223 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beef9ba-d06c-486c-8187-b5ee550a4ffb" containerName="manager" Jul 29 07:22:34.944296 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.944241 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beef9ba-d06c-486c-8187-b5ee550a4ffb" containerName="manager" Jul 29 07:22:34.944331 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.944306 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beef9ba-d06c-486c-8187-b5ee550a4ffb" containerName="manager" Jul 29 07:22:34.946387 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.946368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:34.949356 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.949316 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2l2dr\"" Jul 29 07:22:34.949356 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.949316 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 07:22:34.957709 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:34.957681 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:35.044266 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.044203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:35.054181 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.054148 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:35.107867 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.107833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfvz\" (UniqueName: \"kubernetes.io/projected/911e17f2-8889-456b-a56d-c255359238b5-kube-api-access-hgfvz\") pod \"limitador-limitador-64c94fd64f-wkfxs\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.108090 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.107908 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/911e17f2-8889-456b-a56d-c255359238b5-config-file\") pod \"limitador-limitador-64c94fd64f-wkfxs\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.118283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.118242 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:35.208726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.208631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/911e17f2-8889-456b-a56d-c255359238b5-config-file\") pod \"limitador-limitador-64c94fd64f-wkfxs\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.208726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.208700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfvz\" (UniqueName: \"kubernetes.io/projected/911e17f2-8889-456b-a56d-c255359238b5-kube-api-access-hgfvz\") pod \"limitador-limitador-64c94fd64f-wkfxs\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.209332 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.209313 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/911e17f2-8889-456b-a56d-c255359238b5-config-file\") pod \"limitador-limitador-64c94fd64f-wkfxs\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.217928 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.217894 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfvz\" (UniqueName: \"kubernetes.io/projected/911e17f2-8889-456b-a56d-c255359238b5-kube-api-access-hgfvz\") pod \"limitador-limitador-64c94fd64f-wkfxs\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.258881 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.258844 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:35.414846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.414813 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:35.419181 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:22:35.419149 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911e17f2_8889_456b_a56d_c255359238b5.slice/crio-c87758f24f981a046aff08c3918046e1889d6f6872da18265984da734a82b93c WatchSource:0}: Error finding container c87758f24f981a046aff08c3918046e1889d6f6872da18265984da734a82b93c: Status 404 returned error can't find the container with id c87758f24f981a046aff08c3918046e1889d6f6872da18265984da734a82b93c Jul 29 07:22:35.787743 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.787682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:35.888985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.888942 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:35.931729 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.931690 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-mldsc"] Jul 29 07:22:35.934859 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.934837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:22:35.937513 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.937489 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qw5qb\"" Jul 29 07:22:35.941121 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:35.941094 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mldsc"] Jul 29 07:22:36.017138 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:36.017097 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktblx\" (UniqueName: \"kubernetes.io/projected/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2-kube-api-access-ktblx\") pod \"authorino-75d556bd77-mldsc\" (UID: \"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2\") " pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:22:36.118905 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:36.118666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktblx\" (UniqueName: \"kubernetes.io/projected/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2-kube-api-access-ktblx\") pod \"authorino-75d556bd77-mldsc\" (UID: \"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2\") " pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:22:36.130238 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:36.130204 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktblx\" (UniqueName: \"kubernetes.io/projected/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2-kube-api-access-ktblx\") pod \"authorino-75d556bd77-mldsc\" (UID: \"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2\") " pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:22:36.246297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:36.246256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:22:36.282419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:36.282329 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" event={"ID":"911e17f2-8889-456b-a56d-c255359238b5","Type":"ContainerStarted","Data":"c87758f24f981a046aff08c3918046e1889d6f6872da18265984da734a82b93c"} Jul 29 07:22:36.509073 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:36.507070 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mldsc"] Jul 29 07:22:37.289842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:37.289807 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mldsc" event={"ID":"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2","Type":"ContainerStarted","Data":"090c40edada82c6ea641777355b93752393619d4f95dbcb685b7e93465d04f9c"} Jul 29 07:22:39.308066 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:39.308002 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" event={"ID":"911e17f2-8889-456b-a56d-c255359238b5","Type":"ContainerStarted","Data":"133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba"} Jul 29 07:22:39.308511 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:39.308096 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:39.327244 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:39.327173 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" podStartSLOduration=2.421395767 podStartE2EDuration="5.327150592s" totalImagesPullingTime="2.905754825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:22:39.325151525 +0000 UTC m=+469.096602326" watchObservedRunningTime="2026-07-29 07:22:39.327150592 +0000 UTC m=+469.098601396" Jul 29 07:22:41.321974 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:41.321937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mldsc" event={"ID":"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2","Type":"ContainerStarted","Data":"1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca"} Jul 29 07:22:41.338694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:41.338633 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-mldsc" podStartSLOduration=2.462429487 podStartE2EDuration="6.338614921s" totalImagesPullingTime="3.876185434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:22:41.336827208 +0000 UTC m=+471.108278008" watchObservedRunningTime="2026-07-29 07:22:41.338614921 +0000 UTC m=+471.110065720" Jul 29 07:22:50.315914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:50.315878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:50.649567 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:50.649481 2568 scope.go:117] "RemoveContainer" containerID="989716c09a9b9de6d3f44d35ec604afa275c692eebe48125775180cff2b7f80c" Jul 29 07:22:51.745998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:51.745961 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:51.746399 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:51.746157 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" podUID="911e17f2-8889-456b-a56d-c255359238b5" containerName="limitador" containerID="cri-o://133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba" gracePeriod=30 Jul 29 07:22:52.301580 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.301551 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:52.357520 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.357423 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgfvz\" (UniqueName: \"kubernetes.io/projected/911e17f2-8889-456b-a56d-c255359238b5-kube-api-access-hgfvz\") pod \"911e17f2-8889-456b-a56d-c255359238b5\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " Jul 29 07:22:52.357701 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.357524 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/911e17f2-8889-456b-a56d-c255359238b5-config-file\") pod \"911e17f2-8889-456b-a56d-c255359238b5\" (UID: \"911e17f2-8889-456b-a56d-c255359238b5\") " Jul 29 07:22:52.357893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.357872 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911e17f2-8889-456b-a56d-c255359238b5-config-file" (OuterVolumeSpecName: "config-file") pod "911e17f2-8889-456b-a56d-c255359238b5" (UID: "911e17f2-8889-456b-a56d-c255359238b5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:52.359719 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.359691 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911e17f2-8889-456b-a56d-c255359238b5-kube-api-access-hgfvz" (OuterVolumeSpecName: "kube-api-access-hgfvz") pod "911e17f2-8889-456b-a56d-c255359238b5" (UID: "911e17f2-8889-456b-a56d-c255359238b5"). InnerVolumeSpecName "kube-api-access-hgfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:52.377714 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.377678 2568 generic.go:356] "Generic (PLEG): container finished" podID="911e17f2-8889-456b-a56d-c255359238b5" containerID="133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba" exitCode=0 Jul 29 07:22:52.377883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.377730 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" Jul 29 07:22:52.377883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.377764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" event={"ID":"911e17f2-8889-456b-a56d-c255359238b5","Type":"ContainerDied","Data":"133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba"} Jul 29 07:22:52.377883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.377823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wkfxs" event={"ID":"911e17f2-8889-456b-a56d-c255359238b5","Type":"ContainerDied","Data":"c87758f24f981a046aff08c3918046e1889d6f6872da18265984da734a82b93c"} Jul 29 07:22:52.377883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.377847 2568 scope.go:117] "RemoveContainer" containerID="133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba" Jul 29 07:22:52.387334 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.387310 2568 scope.go:117] "RemoveContainer" containerID="133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba" Jul 29 07:22:52.387684 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:22:52.387666 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba\": container with ID starting with 133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba not found: ID does not exist" containerID="133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba" Jul 29 07:22:52.387724 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.387695 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba"} err="failed to get container status \"133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba\": rpc error: code = NotFound desc = could not find container \"133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba\": container with ID starting with 133646b7853026a67037780b2c3efc579057cfaf526a1572a589610eaf4946ba not found: ID does not exist" Jul 29 07:22:52.401942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.401907 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:52.406515 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.406485 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wkfxs"] Jul 29 07:22:52.458839 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.458798 2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/911e17f2-8889-456b-a56d-c255359238b5-config-file\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:22:52.458839 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.458830 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfvz\" (UniqueName: \"kubernetes.io/projected/911e17f2-8889-456b-a56d-c255359238b5-kube-api-access-hgfvz\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:22:52.741966 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:52.741932 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911e17f2-8889-456b-a56d-c255359238b5" path="/var/lib/kubelet/pods/911e17f2-8889-456b-a56d-c255359238b5/volumes" Jul 29 07:22:56.718018 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:56.717979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:56.816595 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:56.816547 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:57.155189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.155150 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7"] Jul 29 07:22:57.156062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.156019 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911e17f2-8889-456b-a56d-c255359238b5" containerName="limitador" Jul 29 07:22:57.156238 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.156225 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="911e17f2-8889-456b-a56d-c255359238b5" containerName="limitador" Jul 29 07:22:57.156439 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.156426 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="911e17f2-8889-456b-a56d-c255359238b5" containerName="limitador" Jul 29 07:22:57.159598 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.159572 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.163442 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.163416 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 07:22:57.163442 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.163438 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:22:57.164600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.164583 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ngrfr\"" Jul 29 07:22:57.164683 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.164598 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:22:57.176174 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.176138 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7"] Jul 29 07:22:57.200597 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.200551 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:57.203637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.203607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9557216-c2c5-4946-80e0-fdc3c620c33e-data\") pod \"postgres-69fb4594cf-gcbw7\" (UID: \"d9557216-c2c5-4946-80e0-fdc3c620c33e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.203823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.203651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv8q8\" (UniqueName: \"kubernetes.io/projected/d9557216-c2c5-4946-80e0-fdc3c620c33e-kube-api-access-fv8q8\") pod \"postgres-69fb4594cf-gcbw7\" (UID: \"d9557216-c2c5-4946-80e0-fdc3c620c33e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.305001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.304954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9557216-c2c5-4946-80e0-fdc3c620c33e-data\") pod \"postgres-69fb4594cf-gcbw7\" (UID: \"d9557216-c2c5-4946-80e0-fdc3c620c33e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.305211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.305016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8q8\" (UniqueName: \"kubernetes.io/projected/d9557216-c2c5-4946-80e0-fdc3c620c33e-kube-api-access-fv8q8\") pod \"postgres-69fb4594cf-gcbw7\" (UID: \"d9557216-c2c5-4946-80e0-fdc3c620c33e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.305403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.305381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9557216-c2c5-4946-80e0-fdc3c620c33e-data\") pod \"postgres-69fb4594cf-gcbw7\" (UID: \"d9557216-c2c5-4946-80e0-fdc3c620c33e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.314077 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.314021 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8q8\" (UniqueName: \"kubernetes.io/projected/d9557216-c2c5-4946-80e0-fdc3c620c33e-kube-api-access-fv8q8\") pod \"postgres-69fb4594cf-gcbw7\" (UID: \"d9557216-c2c5-4946-80e0-fdc3c620c33e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.472905 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.472808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:22:57.622341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:57.622309 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7"] Jul 29 07:22:57.624126 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:22:57.624096 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9557216_c2c5_4946_80e0_fdc3c620c33e.slice/crio-5651fc5592715524206aea07c5bd7c653884dfbc699bcbcf9e05f3697d2e1964 WatchSource:0}: Error finding container 5651fc5592715524206aea07c5bd7c653884dfbc699bcbcf9e05f3697d2e1964: Status 404 returned error can't find the container with id 5651fc5592715524206aea07c5bd7c653884dfbc699bcbcf9e05f3697d2e1964 Jul 29 07:22:58.415934 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:22:58.415893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" event={"ID":"d9557216-c2c5-4946-80e0-fdc3c620c33e","Type":"ContainerStarted","Data":"5651fc5592715524206aea07c5bd7c653884dfbc699bcbcf9e05f3697d2e1964"} Jul 29 07:23:01.876606 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:01.876563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:03.446428 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:03.446391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" event={"ID":"d9557216-c2c5-4946-80e0-fdc3c620c33e","Type":"ContainerStarted","Data":"e9a48d1abd0256f7b17453fb4e3444d6d5c3f72f9c13b5ce5bd9766a68289f7e"} Jul 29 07:23:03.446966 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:03.446506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:23:03.475457 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:03.475376 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" podStartSLOduration=1.5776188389999999 podStartE2EDuration="6.475357432s" totalImagesPullingTime="4.897738593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:03.473314855 +0000 UTC m=+493.244765655" watchObservedRunningTime="2026-07-29 07:23:03.475357432 +0000 UTC m=+493.246808230" Jul 29 07:23:09.368753 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:09.368713 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:09.483374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:09.483342 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gcbw7" Jul 29 07:23:09.922850 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:09.922804 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:10.106119 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.106080 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:10.199204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.199092 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j7mhn"] Jul 29 07:23:10.204603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.204574 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:10.223225 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.223182 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j7mhn"] Jul 29 07:23:10.331826 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.331785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbzq\" (UniqueName: \"kubernetes.io/projected/0eaa76bb-c254-41b5-9120-7a5fa233a19f-kube-api-access-6nbzq\") pod \"authorino-5d4b67c446-j7mhn\" (UID: \"0eaa76bb-c254-41b5-9120-7a5fa233a19f\") " pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:10.432463 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.432421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbzq\" (UniqueName: \"kubernetes.io/projected/0eaa76bb-c254-41b5-9120-7a5fa233a19f-kube-api-access-6nbzq\") pod \"authorino-5d4b67c446-j7mhn\" (UID: \"0eaa76bb-c254-41b5-9120-7a5fa233a19f\") " pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:10.459000 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.458915 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbzq\" (UniqueName: \"kubernetes.io/projected/0eaa76bb-c254-41b5-9120-7a5fa233a19f-kube-api-access-6nbzq\") pod \"authorino-5d4b67c446-j7mhn\" (UID: \"0eaa76bb-c254-41b5-9120-7a5fa233a19f\") " pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:10.491381 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.491347 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j7mhn"] Jul 29 07:23:10.491625 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.491612 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:10.577583 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.576531 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-679959c66-4lvhq"] Jul 29 07:23:10.579620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.579594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:10.605157 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.605093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-679959c66-4lvhq"] Jul 29 07:23:10.663533 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.663500 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j7mhn"] Jul 29 07:23:10.667445 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:23:10.667416 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eaa76bb_c254_41b5_9120_7a5fa233a19f.slice/crio-547224e632b4061dc48cd686acbd60816d4d6f9dcb09f5812b6d955c7fcb2274 WatchSource:0}: Error finding container 547224e632b4061dc48cd686acbd60816d4d6f9dcb09f5812b6d955c7fcb2274: Status 404 returned error can't find the container with id 547224e632b4061dc48cd686acbd60816d4d6f9dcb09f5812b6d955c7fcb2274 Jul 29 07:23:10.734722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.734636 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j667j\" (UniqueName: \"kubernetes.io/projected/11c3ab70-d0b0-477c-8830-48c97f62b761-kube-api-access-j667j\") pod \"authorino-679959c66-4lvhq\" (UID: \"11c3ab70-d0b0-477c-8830-48c97f62b761\") " pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:10.774885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.774843 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-679959c66-4lvhq"] Jul 29 07:23:10.775157 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:23:10.775136 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j667j], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-679959c66-4lvhq" podUID="11c3ab70-d0b0-477c-8830-48c97f62b761" Jul 29 07:23:10.834797 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.834757 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56b77dd99d-hgtd2"] Jul 29 07:23:10.835105 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.835074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j667j\" (UniqueName: \"kubernetes.io/projected/11c3ab70-d0b0-477c-8830-48c97f62b761-kube-api-access-j667j\") pod \"authorino-679959c66-4lvhq\" (UID: \"11c3ab70-d0b0-477c-8830-48c97f62b761\") " pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:10.837437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.837416 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:10.863276 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.863232 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 07:23:10.877840 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.877805 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b77dd99d-hgtd2"] Jul 29 07:23:10.884718 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.884687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j667j\" (UniqueName: \"kubernetes.io/projected/11c3ab70-d0b0-477c-8830-48c97f62b761-kube-api-access-j667j\") pod \"authorino-679959c66-4lvhq\" (UID: \"11c3ab70-d0b0-477c-8830-48c97f62b761\") " pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:10.936538 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.936498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qfv\" (UniqueName: \"kubernetes.io/projected/272a44b0-5601-4845-a8f3-d8de76d612c7-kube-api-access-54qfv\") pod \"authorino-56b77dd99d-hgtd2\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:10.936729 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:10.936565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/272a44b0-5601-4845-a8f3-d8de76d612c7-tls-cert\") pod \"authorino-56b77dd99d-hgtd2\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:11.037392 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.037294 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qfv\" (UniqueName: \"kubernetes.io/projected/272a44b0-5601-4845-a8f3-d8de76d612c7-kube-api-access-54qfv\") pod \"authorino-56b77dd99d-hgtd2\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:11.037573 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.037394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/272a44b0-5601-4845-a8f3-d8de76d612c7-tls-cert\") pod \"authorino-56b77dd99d-hgtd2\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:11.040188 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.040150 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/272a44b0-5601-4845-a8f3-d8de76d612c7-tls-cert\") pod \"authorino-56b77dd99d-hgtd2\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:11.053851 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.053824 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qfv\" (UniqueName: \"kubernetes.io/projected/272a44b0-5601-4845-a8f3-d8de76d612c7-kube-api-access-54qfv\") pod \"authorino-56b77dd99d-hgtd2\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:11.148212 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.148174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:23:11.326140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.326108 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b77dd99d-hgtd2"] Jul 29 07:23:11.326586 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:23:11.326561 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod272a44b0_5601_4845_a8f3_d8de76d612c7.slice/crio-48d05c16882065365faebcdd4313513dee511dacf04d01d91c242080743faebc WatchSource:0}: Error finding container 48d05c16882065365faebcdd4313513dee511dacf04d01d91c242080743faebc: Status 404 returned error can't find the container with id 48d05c16882065365faebcdd4313513dee511dacf04d01d91c242080743faebc Jul 29 07:23:11.487520 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.487470 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" event={"ID":"272a44b0-5601-4845-a8f3-d8de76d612c7","Type":"ContainerStarted","Data":"48d05c16882065365faebcdd4313513dee511dacf04d01d91c242080743faebc"} Jul 29 07:23:11.488985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.488960 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:11.488985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.488971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" event={"ID":"0eaa76bb-c254-41b5-9120-7a5fa233a19f","Type":"ContainerStarted","Data":"226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae"} Jul 29 07:23:11.489167 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.489007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" event={"ID":"0eaa76bb-c254-41b5-9120-7a5fa233a19f","Type":"ContainerStarted","Data":"547224e632b4061dc48cd686acbd60816d4d6f9dcb09f5812b6d955c7fcb2274"} Jul 29 07:23:11.489167 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.489021 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" podUID="0eaa76bb-c254-41b5-9120-7a5fa233a19f" containerName="authorino" containerID="cri-o://226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae" gracePeriod=30 Jul 29 07:23:11.496436 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.496408 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:11.543150 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.543090 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" podStartSLOduration=1.1631713399999999 podStartE2EDuration="1.543070545s" totalImagesPullingTime="379.899205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:11.541560096 +0000 UTC m=+501.313010897" watchObservedRunningTime="2026-07-29 07:23:11.543070545 +0000 UTC m=+501.314521344" Jul 29 07:23:11.642960 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.642924 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j667j\" (UniqueName: \"kubernetes.io/projected/11c3ab70-d0b0-477c-8830-48c97f62b761-kube-api-access-j667j\") pod \"11c3ab70-d0b0-477c-8830-48c97f62b761\" (UID: \"11c3ab70-d0b0-477c-8830-48c97f62b761\") " Jul 29 07:23:11.645339 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.645302 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c3ab70-d0b0-477c-8830-48c97f62b761-kube-api-access-j667j" (OuterVolumeSpecName: "kube-api-access-j667j") pod "11c3ab70-d0b0-477c-8830-48c97f62b761" (UID: "11c3ab70-d0b0-477c-8830-48c97f62b761"). InnerVolumeSpecName "kube-api-access-j667j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:23:11.744404 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.744367 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j667j\" (UniqueName: \"kubernetes.io/projected/11c3ab70-d0b0-477c-8830-48c97f62b761-kube-api-access-j667j\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:23:11.803560 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.803530 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:11.945696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.945666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbzq\" (UniqueName: \"kubernetes.io/projected/0eaa76bb-c254-41b5-9120-7a5fa233a19f-kube-api-access-6nbzq\") pod \"0eaa76bb-c254-41b5-9120-7a5fa233a19f\" (UID: \"0eaa76bb-c254-41b5-9120-7a5fa233a19f\") " Jul 29 07:23:11.948210 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:11.948177 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaa76bb-c254-41b5-9120-7a5fa233a19f-kube-api-access-6nbzq" (OuterVolumeSpecName: "kube-api-access-6nbzq") pod "0eaa76bb-c254-41b5-9120-7a5fa233a19f" (UID: "0eaa76bb-c254-41b5-9120-7a5fa233a19f"). InnerVolumeSpecName "kube-api-access-6nbzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:23:12.046453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.046406 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbzq\" (UniqueName: \"kubernetes.io/projected/0eaa76bb-c254-41b5-9120-7a5fa233a19f-kube-api-access-6nbzq\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:23:12.496284 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.496245 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" event={"ID":"272a44b0-5601-4845-a8f3-d8de76d612c7","Type":"ContainerStarted","Data":"73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318"} Jul 29 07:23:12.497675 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.497646 2568 generic.go:356] "Generic (PLEG): container finished" podID="0eaa76bb-c254-41b5-9120-7a5fa233a19f" containerID="226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae" exitCode=0 Jul 29 07:23:12.497822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.497682 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" Jul 29 07:23:12.497822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.497696 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-679959c66-4lvhq" Jul 29 07:23:12.497822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.497721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" event={"ID":"0eaa76bb-c254-41b5-9120-7a5fa233a19f","Type":"ContainerDied","Data":"226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae"} Jul 29 07:23:12.497822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.497750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-j7mhn" event={"ID":"0eaa76bb-c254-41b5-9120-7a5fa233a19f","Type":"ContainerDied","Data":"547224e632b4061dc48cd686acbd60816d4d6f9dcb09f5812b6d955c7fcb2274"} Jul 29 07:23:12.497822 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.497766 2568 scope.go:117] "RemoveContainer" containerID="226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae" Jul 29 07:23:12.507672 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.507652 2568 scope.go:117] "RemoveContainer" containerID="226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae" Jul 29 07:23:12.508030 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:23:12.508009 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae\": container with ID starting with 226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae not found: ID does not exist" containerID="226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae" Jul 29 07:23:12.508107 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.508055 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae"} err="failed to get container status \"226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae\": rpc error: code = NotFound desc = could not find container \"226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae\": container with ID starting with 226e2b7fc7027ab942cb7217c6328fce634c1a37d458f43ed82664c5a7dc26ae not found: ID does not exist" Jul 29 07:23:12.524433 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.524381 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" podStartSLOduration=2.155031607 podStartE2EDuration="2.524361229s" totalImagesPullingTime="369.329622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:12.518539577 +0000 UTC m=+502.289990378" watchObservedRunningTime="2026-07-29 07:23:12.524361229 +0000 UTC m=+502.295812027" Jul 29 07:23:12.566335 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.566292 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j7mhn"] Jul 29 07:23:12.582466 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.582421 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mldsc"] Jul 29 07:23:12.582659 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.582626 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-mldsc" podUID="0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" containerName="authorino" containerID="cri-o://1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca" gracePeriod=30 Jul 29 07:23:12.584652 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.584618 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j7mhn"] Jul 29 07:23:12.644612 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.644570 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-679959c66-4lvhq"] Jul 29 07:23:12.646912 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.646879 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-679959c66-4lvhq"] Jul 29 07:23:12.738766 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.738682 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eaa76bb-c254-41b5-9120-7a5fa233a19f" path="/var/lib/kubelet/pods/0eaa76bb-c254-41b5-9120-7a5fa233a19f/volumes" Jul 29 07:23:12.739017 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.739003 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c3ab70-d0b0-477c-8830-48c97f62b761" path="/var/lib/kubelet/pods/11c3ab70-d0b0-477c-8830-48c97f62b761/volumes" Jul 29 07:23:12.847931 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.847901 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:23:12.953856 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.953814 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktblx\" (UniqueName: \"kubernetes.io/projected/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2-kube-api-access-ktblx\") pod \"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2\" (UID: \"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2\") " Jul 29 07:23:12.956135 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:12.956106 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2-kube-api-access-ktblx" (OuterVolumeSpecName: "kube-api-access-ktblx") pod "0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" (UID: "0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2"). InnerVolumeSpecName "kube-api-access-ktblx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:23:13.054626 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.054530 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktblx\" (UniqueName: \"kubernetes.io/projected/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2-kube-api-access-ktblx\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:23:13.504403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.504359 2568 generic.go:356] "Generic (PLEG): container finished" podID="0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" containerID="1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca" exitCode=0 Jul 29 07:23:13.504403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.504403 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mldsc" Jul 29 07:23:13.504966 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.504443 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mldsc" event={"ID":"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2","Type":"ContainerDied","Data":"1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca"} Jul 29 07:23:13.504966 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.504496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mldsc" event={"ID":"0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2","Type":"ContainerDied","Data":"090c40edada82c6ea641777355b93752393619d4f95dbcb685b7e93465d04f9c"} Jul 29 07:23:13.504966 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.504524 2568 scope.go:117] "RemoveContainer" containerID="1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca" Jul 29 07:23:13.514532 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.514511 2568 scope.go:117] "RemoveContainer" containerID="1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca" Jul 29 07:23:13.514877 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:23:13.514855 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca\": container with ID starting with 1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca not found: ID does not exist" containerID="1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca" Jul 29 07:23:13.514943 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.514889 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca"} err="failed to get container status \"1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca\": rpc error: code = NotFound desc = could not find container \"1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca\": container with ID starting with 1810e649aedc9337df477934cd293336c3cd2f0e0b0787286f5eb126a0aa28ca not found: ID does not exist" Jul 29 07:23:13.544791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.544755 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mldsc"] Jul 29 07:23:13.547357 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:13.547325 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mldsc"] Jul 29 07:23:14.744091 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:14.744028 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" path="/var/lib/kubelet/pods/0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2/volumes" Jul 29 07:23:17.799324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:17.799288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:17.959969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:17.959929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:18.171693 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.171659 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-664896b984-x9mcn"] Jul 29 07:23:18.172160 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172123 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:18.172314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172239 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" containerName="authorino" Jul 29 07:23:18.172314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172254 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" containerName="authorino" Jul 29 07:23:18.172314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172266 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eaa76bb-c254-41b5-9120-7a5fa233a19f" containerName="authorino" Jul 29 07:23:18.172314 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172272 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eaa76bb-c254-41b5-9120-7a5fa233a19f" containerName="authorino" Jul 29 07:23:18.172456 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172339 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fcd67bd-88e4-4e5e-b27e-b0ab8831b8a2" containerName="authorino" Jul 29 07:23:18.172456 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.172348 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eaa76bb-c254-41b5-9120-7a5fa233a19f" containerName="authorino" Jul 29 07:23:18.178666 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.178633 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.195343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.195167 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 07:23:18.195343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.195184 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qpfx2\"" Jul 29 07:23:18.195343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.195240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Jul 29 07:23:18.195343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.195263 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 07:23:18.197524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.197499 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664896b984-x9mcn"] Jul 29 07:23:18.303362 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.303318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp2c\" (UniqueName: \"kubernetes.io/projected/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-kube-api-access-lhp2c\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.303362 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.303363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-webhook-cert\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.303587 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.303441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-metrics-certs\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.404923 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.404878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp2c\" (UniqueName: \"kubernetes.io/projected/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-kube-api-access-lhp2c\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.404923 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.404927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-webhook-cert\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.405261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.404960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-metrics-certs\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.407888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.407857 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-webhook-cert\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.407983 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.407922 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-metrics-certs\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.417097 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.417065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp2c\" (UniqueName: \"kubernetes.io/projected/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-kube-api-access-lhp2c\") pod \"maas-controller-664896b984-x9mcn\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.490115 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.490005 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:18.641623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:18.641593 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664896b984-x9mcn"] Jul 29 07:23:18.643845 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:23:18.643816 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64adbfd0_cb9b_4680_8893_95c6cc7ae44e.slice/crio-64395661a30410eaa5df6526805c09be24d259051cea5c3106ddae67076bc795 WatchSource:0}: Error finding container 64395661a30410eaa5df6526805c09be24d259051cea5c3106ddae67076bc795: Status 404 returned error can't find the container with id 64395661a30410eaa5df6526805c09be24d259051cea5c3106ddae67076bc795 Jul 29 07:23:19.539988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:19.539952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-x9mcn" event={"ID":"64adbfd0-cb9b-4680-8893-95c6cc7ae44e","Type":"ContainerStarted","Data":"64395661a30410eaa5df6526805c09be24d259051cea5c3106ddae67076bc795"} Jul 29 07:23:20.956432 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:20.956395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:21.062937 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:21.062905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:21.555673 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:21.555619 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-x9mcn" event={"ID":"64adbfd0-cb9b-4680-8893-95c6cc7ae44e","Type":"ContainerStarted","Data":"abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f"} Jul 29 07:23:21.555897 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:21.555811 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:21.594736 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:21.594678 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-664896b984-x9mcn" podStartSLOduration=1.357116806 podStartE2EDuration="3.594657495s" totalImagesPullingTime="2.237540689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:21.593170785 +0000 UTC m=+511.364621596" watchObservedRunningTime="2026-07-29 07:23:21.594657495 +0000 UTC m=+511.366108294" Jul 29 07:23:32.563124 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:32.563088 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:23:52.883111 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:52.883057 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:53.000156 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.000112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:53.481678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.481635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:53.615235 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.615198 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f"] Jul 29 07:23:53.620652 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.620620 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.623678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.623646 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 07:23:53.623892 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.623878 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:23:53.662034 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.661982 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f"] Jul 29 07:23:53.721560 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.721525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gk7\" (UniqueName: \"kubernetes.io/projected/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-kube-api-access-w8gk7\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.721771 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.721594 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-openshift-service-ca\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.721771 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.721672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-maas-api-tls\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.822453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.822355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gk7\" (UniqueName: \"kubernetes.io/projected/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-kube-api-access-w8gk7\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.822453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.822430 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-openshift-service-ca\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.822453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.822453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-maas-api-tls\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.823172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.823151 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-openshift-service-ca\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.825101 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.825077 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-maas-api-tls\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.840248 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.840217 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gk7\" (UniqueName: \"kubernetes.io/projected/6c163dde-8e3c-43a1-8dc7-7a2176ad513d-kube-api-access-w8gk7\") pod \"maas-api-7d46f59bcc-x5b6f\" (UID: \"6c163dde-8e3c-43a1-8dc7-7a2176ad513d\") " pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.932415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.932376 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:53.965480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:53.965444 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:54.092062 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:54.091953 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f"] Jul 29 07:23:54.120820 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:23:54.120785 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c163dde_8e3c_43a1_8dc7_7a2176ad513d.slice/crio-ba4a16e38077105a377fede901a041ef144bea08374742716153968fc2e86ea5 WatchSource:0}: Error finding container ba4a16e38077105a377fede901a041ef144bea08374742716153968fc2e86ea5: Status 404 returned error can't find the container with id ba4a16e38077105a377fede901a041ef144bea08374742716153968fc2e86ea5 Jul 29 07:23:54.726516 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:54.726480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" event={"ID":"6c163dde-8e3c-43a1-8dc7-7a2176ad513d","Type":"ContainerStarted","Data":"ba4a16e38077105a377fede901a041ef144bea08374742716153968fc2e86ea5"} Jul 29 07:23:54.772587 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:54.772544 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:55.033324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:55.033240 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:56.739462 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:56.739428 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:23:56.739863 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:56.739481 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" event={"ID":"6c163dde-8e3c-43a1-8dc7-7a2176ad513d","Type":"ContainerStarted","Data":"3c0eeb53c0d596128f02d609a6a4477d89012685f2f28d6eac5fbd05d99e8021"} Jul 29 07:23:56.774973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:23:56.774914 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" podStartSLOduration=2.200579311 podStartE2EDuration="3.774893919s" totalImagesPullingTime="1.574314608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:56.768455102 +0000 UTC m=+546.539905902" watchObservedRunningTime="2026-07-29 07:23:56.774893919 +0000 UTC m=+546.546344719" Jul 29 07:24:00.877699 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:00.877661 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:24:02.750981 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:02.750952 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d46f59bcc-x5b6f" Jul 29 07:24:04.560735 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:04.560695 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:24:04.711755 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:04.711704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:24:10.150100 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:10.150061 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:24:18.209845 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.209796 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p"] Jul 29 07:24:18.213984 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.213958 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.219293 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.219262 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 07:24:18.225445 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.225415 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 07:24:18.225623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.225465 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4s7sl\"" Jul 29 07:24:18.229266 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.229242 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 07:24:18.255376 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.255339 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p"] Jul 29 07:24:18.289342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.289301 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:24:18.344308 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.344272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.344308 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.344311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.344604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.344336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpg4\" (UniqueName: \"kubernetes.io/projected/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-kube-api-access-7qpg4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.344604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.344447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.344604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.344486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.344604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.344516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445182 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445182 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445473 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpg4\" (UniqueName: \"kubernetes.io/projected/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-kube-api-access-7qpg4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445473 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445473 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445473 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445651 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445703 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445644 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.445742 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.445695 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.448083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.448032 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.448318 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.448298 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.463318 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.463228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpg4\" (UniqueName: \"kubernetes.io/projected/68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc-kube-api-access-7qpg4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s9c4p\" (UID: \"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.526353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.526301 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:18.695333 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.695303 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p"] Jul 29 07:24:18.697824 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:24:18.697785 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68de6b1a_ab33_41a6_86bf_fbb7c0ec59dc.slice/crio-945a669e1b3adca13bdce9ef022bbb8de87b8dcc15b02c2df3d7a71655106d91 WatchSource:0}: Error finding container 945a669e1b3adca13bdce9ef022bbb8de87b8dcc15b02c2df3d7a71655106d91: Status 404 returned error can't find the container with id 945a669e1b3adca13bdce9ef022bbb8de87b8dcc15b02c2df3d7a71655106d91 Jul 29 07:24:18.861614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:18.861524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" event={"ID":"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc","Type":"ContainerStarted","Data":"945a669e1b3adca13bdce9ef022bbb8de87b8dcc15b02c2df3d7a71655106d91"} Jul 29 07:24:20.875794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:20.875753 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" event={"ID":"68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc","Type":"ContainerStarted","Data":"87cfd6004c461e8f6435f136be54f13d784aefd3b34e0c88b6386276b5d0f3ba"} Jul 29 07:24:20.907792 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:20.907733 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" podStartSLOduration=1.77508257 podStartE2EDuration="2.907712826s" totalImagesPullingTime="1.132630256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:24:20.906388229 +0000 UTC m=+570.677839025" watchObservedRunningTime="2026-07-29 07:24:20.907712826 +0000 UTC m=+570.679163625" Jul 29 07:24:28.526633 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:28.526595 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:28.527158 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:28.526668 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:28.539965 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:28.539927 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:28.930484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:28.930453 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s9c4p" Jul 29 07:24:43.325982 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:43.325937 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:24:50.634464 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:50.634430 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:24:50.636875 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:50.636847 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:24:59.930031 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:24:59.929990 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:25:12.501435 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:25:12.501326 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:25:17.733607 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:25:17.733572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:25:37.486772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:25:37.486735 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:25:37.595744 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:25:37.595704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:25:57.646480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:25:57.646437 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:25:59.388853 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:25:59.388817 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:26:11.349888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:11.349841 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:26:13.870370 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:13.870328 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84875955d7-l5s92"] Jul 29 07:26:13.873987 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:13.873960 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:13.891229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:13.891194 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84875955d7-l5s92"] Jul 29 07:26:13.997553 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:13.997511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35d01ecf-5ddc-4d6d-ab32-db348d4540df-tls-cert\") pod \"authorino-84875955d7-l5s92\" (UID: \"35d01ecf-5ddc-4d6d-ab32-db348d4540df\") " pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:13.997553 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:13.997557 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vw6\" (UniqueName: \"kubernetes.io/projected/35d01ecf-5ddc-4d6d-ab32-db348d4540df-kube-api-access-l6vw6\") pod \"authorino-84875955d7-l5s92\" (UID: \"35d01ecf-5ddc-4d6d-ab32-db348d4540df\") " pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:14.098694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.098653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35d01ecf-5ddc-4d6d-ab32-db348d4540df-tls-cert\") pod \"authorino-84875955d7-l5s92\" (UID: \"35d01ecf-5ddc-4d6d-ab32-db348d4540df\") " pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:14.098907 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.098705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vw6\" (UniqueName: \"kubernetes.io/projected/35d01ecf-5ddc-4d6d-ab32-db348d4540df-kube-api-access-l6vw6\") pod \"authorino-84875955d7-l5s92\" (UID: \"35d01ecf-5ddc-4d6d-ab32-db348d4540df\") " pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:14.101433 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.101405 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35d01ecf-5ddc-4d6d-ab32-db348d4540df-tls-cert\") pod \"authorino-84875955d7-l5s92\" (UID: \"35d01ecf-5ddc-4d6d-ab32-db348d4540df\") " pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:14.108108 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.108034 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vw6\" (UniqueName: \"kubernetes.io/projected/35d01ecf-5ddc-4d6d-ab32-db348d4540df-kube-api-access-l6vw6\") pod \"authorino-84875955d7-l5s92\" (UID: \"35d01ecf-5ddc-4d6d-ab32-db348d4540df\") " pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:14.184055 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.184007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84875955d7-l5s92" Jul 29 07:26:14.276782 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.276748 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:26:14.353185 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.353159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84875955d7-l5s92"] Jul 29 07:26:14.356017 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:26:14.355989 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35d01ecf_5ddc_4d6d_ab32_db348d4540df.slice/crio-4c31b63c72f88104e87a111b0eafe29a074c59ddfb9dbcb9573ac278adfe1a8b WatchSource:0}: Error finding container 4c31b63c72f88104e87a111b0eafe29a074c59ddfb9dbcb9573ac278adfe1a8b: Status 404 returned error can't find the container with id 4c31b63c72f88104e87a111b0eafe29a074c59ddfb9dbcb9573ac278adfe1a8b Jul 29 07:26:14.357518 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.357497 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:26:14.392680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.392644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:26:14.482648 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:14.482556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84875955d7-l5s92" event={"ID":"35d01ecf-5ddc-4d6d-ab32-db348d4540df","Type":"ContainerStarted","Data":"4c31b63c72f88104e87a111b0eafe29a074c59ddfb9dbcb9573ac278adfe1a8b"} Jul 29 07:26:15.492509 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.492457 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84875955d7-l5s92" event={"ID":"35d01ecf-5ddc-4d6d-ab32-db348d4540df","Type":"ContainerStarted","Data":"f1bc81735707cd7fd28425b5baccc3bda92a7c9720a5e2ae5bcf28e08678d852"} Jul 29 07:26:15.521017 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.520976 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:26:15.550237 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.550170 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84875955d7-l5s92" podStartSLOduration=1.9190038820000002 podStartE2EDuration="2.550149307s" totalImagesPullingTime="631.145425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:26:15.539948504 +0000 UTC m=+685.311399301" watchObservedRunningTime="2026-07-29 07:26:15.550149307 +0000 UTC m=+685.321600105" Jul 29 07:26:15.626758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.626716 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56b77dd99d-hgtd2"] Jul 29 07:26:15.626964 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.626911 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" podUID="272a44b0-5601-4845-a8f3-d8de76d612c7" containerName="authorino" containerID="cri-o://73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318" gracePeriod=30 Jul 29 07:26:15.681511 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.681469 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:26:15.912443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:15.912413 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:26:16.017707 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.017615 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54qfv\" (UniqueName: \"kubernetes.io/projected/272a44b0-5601-4845-a8f3-d8de76d612c7-kube-api-access-54qfv\") pod \"272a44b0-5601-4845-a8f3-d8de76d612c7\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " Jul 29 07:26:16.017865 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.017723 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/272a44b0-5601-4845-a8f3-d8de76d612c7-tls-cert\") pod \"272a44b0-5601-4845-a8f3-d8de76d612c7\" (UID: \"272a44b0-5601-4845-a8f3-d8de76d612c7\") " Jul 29 07:26:16.020025 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.019989 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272a44b0-5601-4845-a8f3-d8de76d612c7-kube-api-access-54qfv" (OuterVolumeSpecName: "kube-api-access-54qfv") pod "272a44b0-5601-4845-a8f3-d8de76d612c7" (UID: "272a44b0-5601-4845-a8f3-d8de76d612c7"). InnerVolumeSpecName "kube-api-access-54qfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:26:16.030801 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.030758 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272a44b0-5601-4845-a8f3-d8de76d612c7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "272a44b0-5601-4845-a8f3-d8de76d612c7" (UID: "272a44b0-5601-4845-a8f3-d8de76d612c7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:26:16.119014 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.118979 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qfv\" (UniqueName: \"kubernetes.io/projected/272a44b0-5601-4845-a8f3-d8de76d612c7-kube-api-access-54qfv\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:26:16.119014 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.119012 2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/272a44b0-5601-4845-a8f3-d8de76d612c7-tls-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:26:16.499638 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.499605 2568 generic.go:356] "Generic (PLEG): container finished" podID="272a44b0-5601-4845-a8f3-d8de76d612c7" containerID="73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318" exitCode=0 Jul 29 07:26:16.500105 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.499662 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" Jul 29 07:26:16.500105 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.499704 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" event={"ID":"272a44b0-5601-4845-a8f3-d8de76d612c7","Type":"ContainerDied","Data":"73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318"} Jul 29 07:26:16.500105 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.499743 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b77dd99d-hgtd2" event={"ID":"272a44b0-5601-4845-a8f3-d8de76d612c7","Type":"ContainerDied","Data":"48d05c16882065365faebcdd4313513dee511dacf04d01d91c242080743faebc"} Jul 29 07:26:16.500105 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.499760 2568 scope.go:117] "RemoveContainer" containerID="73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318" Jul 29 07:26:16.509393 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.509350 2568 scope.go:117] "RemoveContainer" containerID="73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318" Jul 29 07:26:16.509857 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:26:16.509828 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318\": container with ID starting with 73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318 not found: ID does not exist" containerID="73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318" Jul 29 07:26:16.509990 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.509868 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318"} err="failed to get container status \"73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318\": rpc error: code = NotFound desc = could not find container \"73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318\": container with ID starting with 73b388d0173e02e87e7d47f6c906e4a983b34d76699bb66c122806adf1c4f318 not found: ID does not exist" Jul 29 07:26:16.528605 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.528479 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56b77dd99d-hgtd2"] Jul 29 07:26:16.534773 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.534733 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56b77dd99d-hgtd2"] Jul 29 07:26:16.739528 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:26:16.739483 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272a44b0-5601-4845-a8f3-d8de76d612c7" path="/var/lib/kubelet/pods/272a44b0-5601-4845-a8f3-d8de76d612c7/volumes" Jul 29 07:27:02.759618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:02.759576 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664896b984-x9mcn"] Jul 29 07:27:02.760173 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:02.759800 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-664896b984-x9mcn" podUID="64adbfd0-cb9b-4680-8893-95c6cc7ae44e" containerName="manager" containerID="cri-o://abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f" gracePeriod=30 Jul 29 07:27:03.029666 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.029634 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:27:03.141453 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.141409 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhp2c\" (UniqueName: \"kubernetes.io/projected/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-kube-api-access-lhp2c\") pod \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " Jul 29 07:27:03.141670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.141486 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-webhook-cert\") pod \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " Jul 29 07:27:03.141670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.141525 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-metrics-certs\") pod \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\" (UID: \"64adbfd0-cb9b-4680-8893-95c6cc7ae44e\") " Jul 29 07:27:03.143963 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.143925 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "64adbfd0-cb9b-4680-8893-95c6cc7ae44e" (UID: "64adbfd0-cb9b-4680-8893-95c6cc7ae44e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:27:03.144139 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.143966 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "64adbfd0-cb9b-4680-8893-95c6cc7ae44e" (UID: "64adbfd0-cb9b-4680-8893-95c6cc7ae44e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:27:03.144139 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.143972 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-kube-api-access-lhp2c" (OuterVolumeSpecName: "kube-api-access-lhp2c") pod "64adbfd0-cb9b-4680-8893-95c6cc7ae44e" (UID: "64adbfd0-cb9b-4680-8893-95c6cc7ae44e"). InnerVolumeSpecName "kube-api-access-lhp2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:27:03.242215 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.242173 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp2c\" (UniqueName: \"kubernetes.io/projected/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-kube-api-access-lhp2c\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:27:03.242215 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.242205 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-webhook-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:27:03.242215 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.242214 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64adbfd0-cb9b-4680-8893-95c6cc7ae44e-metrics-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:27:03.777326 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.777287 2568 generic.go:356] "Generic (PLEG): container finished" podID="64adbfd0-cb9b-4680-8893-95c6cc7ae44e" containerID="abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f" exitCode=0 Jul 29 07:27:03.777748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.777338 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-x9mcn" Jul 29 07:27:03.777748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.777388 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-x9mcn" event={"ID":"64adbfd0-cb9b-4680-8893-95c6cc7ae44e","Type":"ContainerDied","Data":"abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f"} Jul 29 07:27:03.777748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.777447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-x9mcn" event={"ID":"64adbfd0-cb9b-4680-8893-95c6cc7ae44e","Type":"ContainerDied","Data":"64395661a30410eaa5df6526805c09be24d259051cea5c3106ddae67076bc795"} Jul 29 07:27:03.777748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.777470 2568 scope.go:117] "RemoveContainer" containerID="abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f" Jul 29 07:27:03.788019 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.787997 2568 scope.go:117] "RemoveContainer" containerID="abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f" Jul 29 07:27:03.788522 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:27:03.788501 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f\": container with ID starting with abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f not found: ID does not exist" containerID="abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f" Jul 29 07:27:03.788602 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.788530 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f"} err="failed to get container status \"abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f\": rpc error: code = NotFound desc = could not find container \"abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f\": container with ID starting with abf8ccd72e6335a3e1101e66c5f5e72a867b440eb88066285590a01eb7e9b94f not found: ID does not exist" Jul 29 07:27:03.801397 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.801355 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664896b984-x9mcn"] Jul 29 07:27:03.805696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:03.805662 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-664896b984-x9mcn"] Jul 29 07:27:04.738596 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:27:04.738558 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64adbfd0-cb9b-4680-8893-95c6cc7ae44e" path="/var/lib/kubelet/pods/64adbfd0-cb9b-4680-8893-95c6cc7ae44e/volumes" Jul 29 07:28:04.649136 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649096 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-664896b984-lm55j"] Jul 29 07:28:04.649634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649536 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272a44b0-5601-4845-a8f3-d8de76d612c7" containerName="authorino" Jul 29 07:28:04.649634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649549 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="272a44b0-5601-4845-a8f3-d8de76d612c7" containerName="authorino" Jul 29 07:28:04.649634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649565 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64adbfd0-cb9b-4680-8893-95c6cc7ae44e" containerName="manager" Jul 29 07:28:04.649634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649570 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64adbfd0-cb9b-4680-8893-95c6cc7ae44e" containerName="manager" Jul 29 07:28:04.649763 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649641 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="64adbfd0-cb9b-4680-8893-95c6cc7ae44e" containerName="manager" Jul 29 07:28:04.649763 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.649651 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="272a44b0-5601-4845-a8f3-d8de76d612c7" containerName="authorino" Jul 29 07:28:04.654179 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.654149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.658461 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.658429 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 07:28:04.658655 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.658499 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qpfx2\"" Jul 29 07:28:04.658655 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.658544 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 07:28:04.658655 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.658503 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Jul 29 07:28:04.662033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.661994 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664896b984-lm55j"] Jul 29 07:28:04.813459 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.813416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-webhook-cert\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.813459 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.813466 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-metrics-certs\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.813720 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.813483 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkt88\" (UniqueName: \"kubernetes.io/projected/2faacfa5-3bea-417a-b310-725f9c6b766f-kube-api-access-bkt88\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.915021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.914990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-webhook-cert\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.915243 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.915037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-metrics-certs\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.915243 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.915094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkt88\" (UniqueName: \"kubernetes.io/projected/2faacfa5-3bea-417a-b310-725f9c6b766f-kube-api-access-bkt88\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.917808 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.917777 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-metrics-certs\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.917991 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.917834 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-webhook-cert\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.925911 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.925873 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkt88\" (UniqueName: \"kubernetes.io/projected/2faacfa5-3bea-417a-b310-725f9c6b766f-kube-api-access-bkt88\") pod \"maas-controller-664896b984-lm55j\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:04.967731 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:04.967683 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:05.128863 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:05.128835 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664896b984-lm55j"] Jul 29 07:28:05.131360 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:28:05.131331 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faacfa5_3bea_417a_b310_725f9c6b766f.slice/crio-7ba5c713251958a97be764a6543ba294242b94297f81e8691a1c4fdb581880bd WatchSource:0}: Error finding container 7ba5c713251958a97be764a6543ba294242b94297f81e8691a1c4fdb581880bd: Status 404 returned error can't find the container with id 7ba5c713251958a97be764a6543ba294242b94297f81e8691a1c4fdb581880bd Jul 29 07:28:06.100891 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:06.100848 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-lm55j" event={"ID":"2faacfa5-3bea-417a-b310-725f9c6b766f","Type":"ContainerStarted","Data":"c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e"} Jul 29 07:28:06.101392 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:06.100912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-lm55j" event={"ID":"2faacfa5-3bea-417a-b310-725f9c6b766f","Type":"ContainerStarted","Data":"7ba5c713251958a97be764a6543ba294242b94297f81e8691a1c4fdb581880bd"} Jul 29 07:28:06.101392 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:06.100940 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:06.125497 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:06.125431 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-664896b984-lm55j" podStartSLOduration=1.72977793 podStartE2EDuration="2.125408464s" totalImagesPullingTime="395.630534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:28:06.124553786 +0000 UTC m=+795.896004586" watchObservedRunningTime="2026-07-29 07:28:06.125408464 +0000 UTC m=+795.896859266" Jul 29 07:28:17.108445 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:17.108360 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:21.813210 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:21.813160 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:28:21.868547 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:21.868504 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664896b984-lm55j"] Jul 29 07:28:21.868760 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:21.868714 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-664896b984-lm55j" podUID="2faacfa5-3bea-417a-b310-725f9c6b766f" containerName="manager" containerID="cri-o://c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e" gracePeriod=30 Jul 29 07:28:22.128922 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.128893 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:22.179624 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.179584 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-metrics-certs\") pod \"2faacfa5-3bea-417a-b310-725f9c6b766f\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " Jul 29 07:28:22.179829 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.179685 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-webhook-cert\") pod \"2faacfa5-3bea-417a-b310-725f9c6b766f\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " Jul 29 07:28:22.179829 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.179744 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkt88\" (UniqueName: \"kubernetes.io/projected/2faacfa5-3bea-417a-b310-725f9c6b766f-kube-api-access-bkt88\") pod \"2faacfa5-3bea-417a-b310-725f9c6b766f\" (UID: \"2faacfa5-3bea-417a-b310-725f9c6b766f\") " Jul 29 07:28:22.182586 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.182535 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2faacfa5-3bea-417a-b310-725f9c6b766f" (UID: "2faacfa5-3bea-417a-b310-725f9c6b766f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:28:22.183378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.183258 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faacfa5-3bea-417a-b310-725f9c6b766f-kube-api-access-bkt88" (OuterVolumeSpecName: "kube-api-access-bkt88") pod "2faacfa5-3bea-417a-b310-725f9c6b766f" (UID: "2faacfa5-3bea-417a-b310-725f9c6b766f"). InnerVolumeSpecName "kube-api-access-bkt88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:28:22.183378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.183347 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2faacfa5-3bea-417a-b310-725f9c6b766f" (UID: "2faacfa5-3bea-417a-b310-725f9c6b766f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:28:22.186501 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.186470 2568 generic.go:356] "Generic (PLEG): container finished" podID="2faacfa5-3bea-417a-b310-725f9c6b766f" containerID="c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e" exitCode=0 Jul 29 07:28:22.186686 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.186531 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-lm55j" Jul 29 07:28:22.186686 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.186553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-lm55j" event={"ID":"2faacfa5-3bea-417a-b310-725f9c6b766f","Type":"ContainerDied","Data":"c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e"} Jul 29 07:28:22.186686 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.186649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-lm55j" event={"ID":"2faacfa5-3bea-417a-b310-725f9c6b766f","Type":"ContainerDied","Data":"7ba5c713251958a97be764a6543ba294242b94297f81e8691a1c4fdb581880bd"} Jul 29 07:28:22.186686 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.186684 2568 scope.go:117] "RemoveContainer" containerID="c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e" Jul 29 07:28:22.205570 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.205550 2568 scope.go:117] "RemoveContainer" containerID="c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e" Jul 29 07:28:22.205926 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:28:22.205907 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e\": container with ID starting with c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e not found: ID does not exist" containerID="c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e" Jul 29 07:28:22.205994 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.205941 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e"} err="failed to get container status \"c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e\": rpc error: code = NotFound desc = could not find container \"c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e\": container with ID starting with c90aa4cb7af8bfa17b6594ec4aaa69182749e4d872883680045bbca5e1ad352e not found: ID does not exist" Jul 29 07:28:22.218058 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.218011 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664896b984-lm55j"] Jul 29 07:28:22.221118 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.221091 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-664896b984-lm55j"] Jul 29 07:28:22.280449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.280406 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-webhook-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:28:22.280449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.280441 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkt88\" (UniqueName: \"kubernetes.io/projected/2faacfa5-3bea-417a-b310-725f9c6b766f-kube-api-access-bkt88\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:28:22.280449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.280451 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2faacfa5-3bea-417a-b310-725f9c6b766f-metrics-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:28:22.738247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.738214 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2faacfa5-3bea-417a-b310-725f9c6b766f" path="/var/lib/kubelet/pods/2faacfa5-3bea-417a-b310-725f9c6b766f/volumes" Jul 29 07:28:22.929183 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:22.929150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:28:23.044635 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.044554 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-664896b984-x29n7"] Jul 29 07:28:23.044945 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.044933 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2faacfa5-3bea-417a-b310-725f9c6b766f" containerName="manager" Jul 29 07:28:23.044997 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.044947 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2faacfa5-3bea-417a-b310-725f9c6b766f" containerName="manager" Jul 29 07:28:23.045030 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.045025 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2faacfa5-3bea-417a-b310-725f9c6b766f" containerName="manager" Jul 29 07:28:23.049407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.049368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.056988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.056675 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qpfx2\"" Jul 29 07:28:23.058309 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.058280 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 07:28:23.058790 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.058580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 07:28:23.058790 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.058698 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Jul 29 07:28:23.060860 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.060836 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664896b984-x29n7"] Jul 29 07:28:23.087236 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.087191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c95489e-a155-46fb-923d-847be73a81dc-webhook-cert\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.087458 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.087313 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c95489e-a155-46fb-923d-847be73a81dc-metrics-certs\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.087458 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.087352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6r9\" (UniqueName: \"kubernetes.io/projected/1c95489e-a155-46fb-923d-847be73a81dc-kube-api-access-vc6r9\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.187828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.187793 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c95489e-a155-46fb-923d-847be73a81dc-metrics-certs\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.188065 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.187842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6r9\" (UniqueName: \"kubernetes.io/projected/1c95489e-a155-46fb-923d-847be73a81dc-kube-api-access-vc6r9\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.188065 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.187930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c95489e-a155-46fb-923d-847be73a81dc-webhook-cert\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.190606 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.190578 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c95489e-a155-46fb-923d-847be73a81dc-webhook-cert\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.190778 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.190697 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c95489e-a155-46fb-923d-847be73a81dc-metrics-certs\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.202713 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.202681 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6r9\" (UniqueName: \"kubernetes.io/projected/1c95489e-a155-46fb-923d-847be73a81dc-kube-api-access-vc6r9\") pod \"maas-controller-664896b984-x29n7\" (UID: \"1c95489e-a155-46fb-923d-847be73a81dc\") " pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.367432 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.367341 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:23.521229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:23.521200 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664896b984-x29n7"] Jul 29 07:28:23.524703 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:28:23.524673 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c95489e_a155_46fb_923d_847be73a81dc.slice/crio-4392a9f1595e0b2a50afcce5b75040e08dbcd1572b3b23b73db3455c2e9a67d4 WatchSource:0}: Error finding container 4392a9f1595e0b2a50afcce5b75040e08dbcd1572b3b23b73db3455c2e9a67d4: Status 404 returned error can't find the container with id 4392a9f1595e0b2a50afcce5b75040e08dbcd1572b3b23b73db3455c2e9a67d4 Jul 29 07:28:24.202541 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:24.202437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-x29n7" event={"ID":"1c95489e-a155-46fb-923d-847be73a81dc","Type":"ContainerStarted","Data":"14b7ca2afa1ac092590cec83d78a515c1c154b4bc52200c801b0c118aeb2f31f"} Jul 29 07:28:24.202541 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:24.202507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664896b984-x29n7" event={"ID":"1c95489e-a155-46fb-923d-847be73a81dc","Type":"ContainerStarted","Data":"4392a9f1595e0b2a50afcce5b75040e08dbcd1572b3b23b73db3455c2e9a67d4"} Jul 29 07:28:24.202997 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:24.202560 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:24.228339 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:24.228280 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-664896b984-x29n7" podStartSLOduration=0.840004149 podStartE2EDuration="1.228258843s" totalImagesPullingTime="388.254694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:28:24.226986197 +0000 UTC m=+813.998436997" watchObservedRunningTime="2026-07-29 07:28:24.228258843 +0000 UTC m=+813.999709641" Jul 29 07:28:35.210622 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:35.210592 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-664896b984-x29n7" Jul 29 07:28:41.649474 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:41.649438 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:28:41.751446 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:28:41.751406 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:11.724352 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:11.724315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:11.820726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:11.820685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:11.989934 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:11.989844 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:14.554600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:14.554562 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:17.975306 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:17.975271 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:18.103847 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:18.103813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:18.221396 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:18.221340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:18.346732 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:18.346654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:24.145059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:24.145017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:24.210494 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:24.210446 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:33.767575 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:33.767537 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:33.827447 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:33.827409 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:46.690670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:46.690580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:29:50.675457 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:50.675426 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:29:50.683724 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:29:50.683700 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:30:00.160167 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.160131 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk"] Jul 29 07:30:00.163417 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.163395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" Jul 29 07:30:00.167195 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.167172 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:30:00.173554 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.173528 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" Jul 29 07:30:00.180802 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.180767 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk"] Jul 29 07:30:00.331142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.331114 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk"] Jul 29 07:30:00.333085 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:30:00.333022 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd022df94_7543_48ce_8101_60213328c804.slice/crio-84f7935f39be89a961e2de0e51d9eb7f5be51bec9a3bab7e6bc2ef67e654bb40 WatchSource:0}: Error finding container 84f7935f39be89a961e2de0e51d9eb7f5be51bec9a3bab7e6bc2ef67e654bb40: Status 404 returned error can't find the container with id 84f7935f39be89a961e2de0e51d9eb7f5be51bec9a3bab7e6bc2ef67e654bb40 Jul 29 07:30:00.715103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:00.715066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" event={"ID":"d022df94-7543-48ce-8101-60213328c804","Type":"ContainerStarted","Data":"84f7935f39be89a961e2de0e51d9eb7f5be51bec9a3bab7e6bc2ef67e654bb40"} Jul 29 07:30:03.734400 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:03.734362 2568 generic.go:356] "Generic (PLEG): container finished" podID="d022df94-7543-48ce-8101-60213328c804" containerID="eaca6a25baf94e91448b4b60ac3c49a50de4cb9fd729597f110403ca11abe060" exitCode=0 Jul 29 07:30:03.734400 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:03.734403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" event={"ID":"d022df94-7543-48ce-8101-60213328c804","Type":"ContainerDied","Data":"eaca6a25baf94e91448b4b60ac3c49a50de4cb9fd729597f110403ca11abe060"} Jul 29 07:30:04.889794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:04.889766 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" Jul 29 07:30:05.745528 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:05.745499 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" Jul 29 07:30:05.745710 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:05.745533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755170-kd9fk" event={"ID":"d022df94-7543-48ce-8101-60213328c804","Type":"ContainerDied","Data":"84f7935f39be89a961e2de0e51d9eb7f5be51bec9a3bab7e6bc2ef67e654bb40"} Jul 29 07:30:05.745710 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:05.745585 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f7935f39be89a961e2de0e51d9eb7f5be51bec9a3bab7e6bc2ef67e654bb40" Jul 29 07:30:08.300847 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:08.300811 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:30:08.401866 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:30:08.401830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:33:40.754200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:40.754163 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27"] Jul 29 07:33:40.754769 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:40.754361 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" podUID="8071ae2d-f735-4489-b7c1-3f3080b9db36" containerName="manager" containerID="cri-o://95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd" gracePeriod=10 Jul 29 07:33:41.346103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.346077 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:33:41.376832 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.376727 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8071ae2d-f735-4489-b7c1-3f3080b9db36-extensions-socket-volume\") pod \"8071ae2d-f735-4489-b7c1-3f3080b9db36\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " Jul 29 07:33:41.376832 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.376788 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6rzq\" (UniqueName: \"kubernetes.io/projected/8071ae2d-f735-4489-b7c1-3f3080b9db36-kube-api-access-s6rzq\") pod \"8071ae2d-f735-4489-b7c1-3f3080b9db36\" (UID: \"8071ae2d-f735-4489-b7c1-3f3080b9db36\") " Jul 29 07:33:41.377249 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.377221 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8071ae2d-f735-4489-b7c1-3f3080b9db36-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8071ae2d-f735-4489-b7c1-3f3080b9db36" (UID: "8071ae2d-f735-4489-b7c1-3f3080b9db36"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:33:41.379780 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.379736 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8071ae2d-f735-4489-b7c1-3f3080b9db36-kube-api-access-s6rzq" (OuterVolumeSpecName: "kube-api-access-s6rzq") pod "8071ae2d-f735-4489-b7c1-3f3080b9db36" (UID: "8071ae2d-f735-4489-b7c1-3f3080b9db36"). InnerVolumeSpecName "kube-api-access-s6rzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:33:41.477834 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.477789 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8071ae2d-f735-4489-b7c1-3f3080b9db36-extensions-socket-volume\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:33:41.477834 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.477822 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rzq\" (UniqueName: \"kubernetes.io/projected/8071ae2d-f735-4489-b7c1-3f3080b9db36-kube-api-access-s6rzq\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:33:41.925332 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.925298 2568 generic.go:356] "Generic (PLEG): container finished" podID="8071ae2d-f735-4489-b7c1-3f3080b9db36" containerID="95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd" exitCode=0 Jul 29 07:33:41.925775 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.925354 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" Jul 29 07:33:41.925775 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.925387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" event={"ID":"8071ae2d-f735-4489-b7c1-3f3080b9db36","Type":"ContainerDied","Data":"95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd"} Jul 29 07:33:41.925775 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.925447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27" event={"ID":"8071ae2d-f735-4489-b7c1-3f3080b9db36","Type":"ContainerDied","Data":"bd059bacc3b5beaddbdf6b573fbfa7e70615bb573467569526c0e7da58558450"} Jul 29 07:33:41.925775 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.925472 2568 scope.go:117] "RemoveContainer" containerID="95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd" Jul 29 07:33:41.936889 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.936869 2568 scope.go:117] "RemoveContainer" containerID="95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd" Jul 29 07:33:41.937309 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:33:41.937286 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd\": container with ID starting with 95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd not found: ID does not exist" containerID="95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd" Jul 29 07:33:41.937380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.937318 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd"} err="failed to get container status \"95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd\": rpc error: code = NotFound desc = could not find container \"95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd\": container with ID starting with 95ff1b5c47c0e6019d4b68c085f31caa0f251d107cd18c3530a032b5b5a165cd not found: ID does not exist" Jul 29 07:33:41.949338 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.949299 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27"] Jul 29 07:33:41.957297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:41.957152 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bbm27"] Jul 29 07:33:42.738566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:33:42.738529 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8071ae2d-f735-4489-b7c1-3f3080b9db36" path="/var/lib/kubelet/pods/8071ae2d-f735-4489-b7c1-3f3080b9db36/volumes" Jul 29 07:34:24.556095 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:24.556031 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:34:26.411302 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:26.411267 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:34:41.949058 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949011 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9"] Jul 29 07:34:41.949498 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949446 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d022df94-7543-48ce-8101-60213328c804" containerName="cleanup" Jul 29 07:34:41.949498 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949459 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d022df94-7543-48ce-8101-60213328c804" containerName="cleanup" Jul 29 07:34:41.949498 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949468 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8071ae2d-f735-4489-b7c1-3f3080b9db36" containerName="manager" Jul 29 07:34:41.949498 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949474 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8071ae2d-f735-4489-b7c1-3f3080b9db36" containerName="manager" Jul 29 07:34:41.949634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949549 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d022df94-7543-48ce-8101-60213328c804" containerName="cleanup" Jul 29 07:34:41.949634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.949558 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8071ae2d-f735-4489-b7c1-3f3080b9db36" containerName="manager" Jul 29 07:34:41.952813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.952782 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:41.956122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.956097 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nbw6s\"" Jul 29 07:34:41.968756 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:41.968724 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9"] Jul 29 07:34:42.016051 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.016004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6696v\" (UniqueName: \"kubernetes.io/projected/89c236b5-5858-47f0-a849-51adee48c8f1-kube-api-access-6696v\") pod \"kuadrant-operator-controller-manager-86878c769f-brgq9\" (UID: \"89c236b5-5858-47f0-a849-51adee48c8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.016234 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.016167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89c236b5-5858-47f0-a849-51adee48c8f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-brgq9\" (UID: \"89c236b5-5858-47f0-a849-51adee48c8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.117675 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.117631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89c236b5-5858-47f0-a849-51adee48c8f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-brgq9\" (UID: \"89c236b5-5858-47f0-a849-51adee48c8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.117866 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.117711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6696v\" (UniqueName: \"kubernetes.io/projected/89c236b5-5858-47f0-a849-51adee48c8f1-kube-api-access-6696v\") pod \"kuadrant-operator-controller-manager-86878c769f-brgq9\" (UID: \"89c236b5-5858-47f0-a849-51adee48c8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.118096 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.118074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89c236b5-5858-47f0-a849-51adee48c8f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-brgq9\" (UID: \"89c236b5-5858-47f0-a849-51adee48c8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.129413 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.129383 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6696v\" (UniqueName: \"kubernetes.io/projected/89c236b5-5858-47f0-a849-51adee48c8f1-kube-api-access-6696v\") pod \"kuadrant-operator-controller-manager-86878c769f-brgq9\" (UID: \"89c236b5-5858-47f0-a849-51adee48c8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.264354 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.264260 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:42.417855 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:42.417823 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9"] Jul 29 07:34:42.420720 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:34:42.420690 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c236b5_5858_47f0_a849_51adee48c8f1.slice/crio-685dea073050e2f72afaed7aa0affd4f6dbfcb4d4f82813f83362be70fb05058 WatchSource:0}: Error finding container 685dea073050e2f72afaed7aa0affd4f6dbfcb4d4f82813f83362be70fb05058: Status 404 returned error can't find the container with id 685dea073050e2f72afaed7aa0affd4f6dbfcb4d4f82813f83362be70fb05058 Jul 29 07:34:43.244860 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:43.244823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" event={"ID":"89c236b5-5858-47f0-a849-51adee48c8f1","Type":"ContainerStarted","Data":"6acb8962dc0b2870509c20363c7b5b51138795e3e5dbca569bf13e0a347091c7"} Jul 29 07:34:43.245328 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:43.244883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" event={"ID":"89c236b5-5858-47f0-a849-51adee48c8f1","Type":"ContainerStarted","Data":"685dea073050e2f72afaed7aa0affd4f6dbfcb4d4f82813f83362be70fb05058"} Jul 29 07:34:43.245328 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:43.244922 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:34:43.270139 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:43.270083 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" podStartSLOduration=2.270036762 podStartE2EDuration="2.270036762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:34:43.268783382 +0000 UTC m=+1193.040234180" watchObservedRunningTime="2026-07-29 07:34:43.270036762 +0000 UTC m=+1193.041487609" Jul 29 07:34:50.705678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:50.705642 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:34:50.715321 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:50.715298 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:34:54.253147 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:34:54.253115 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brgq9" Jul 29 07:37:47.409473 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:37:47.409434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:38:00.655005 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:38:00.654969 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:38:24.762057 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:38:24.762007 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:38:25.884441 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:38:25.884391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:38:48.893259 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:38:48.893170 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:38:49.006766 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:38:49.006727 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:39:04.850220 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:39:04.850169 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:39:50.738161 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:39:50.738129 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:39:50.749813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:39:50.749785 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:40:27.025869 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:40:27.025831 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:40:28.306539 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:40:28.306503 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:41:37.265607 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:41:37.265562 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:41:59.080647 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:41:59.080560 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:02.432671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:02.432627 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:02.557077 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:02.557005 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:02.902079 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:02.902034 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:09.793725 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:09.793686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:09.870174 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:09.870142 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:10.029462 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:10.029427 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:10.534717 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:10.534685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:10.634557 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:10.634522 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:12.540429 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.540391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:12.719467 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.719429 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k"] Jul 29 07:42:12.723351 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.723323 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.727588 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.727558 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cb4ef1f3-serving-cert\"" Jul 29 07:42:12.792432 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.792342 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k"] Jul 29 07:42:12.861524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.861476 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.861524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.861522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-openshift-service-ca\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.861818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.861644 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2vct\" (UniqueName: \"kubernetes.io/projected/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-kube-api-access-s2vct\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.938095 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.938032 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:12.962344 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.962300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.962556 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.962348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-openshift-service-ca\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.962556 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.962424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2vct\" (UniqueName: \"kubernetes.io/projected/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-kube-api-access-s2vct\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.962556 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:42:12.962459 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-serving-cert: secret "maas-api-e2e-ait-cb4ef1f3-serving-cert" not found Jul 29 07:42:12.962556 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:42:12.962521 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls podName:2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf nodeName:}" failed. No retries permitted until 2026-07-29 07:42:13.462504557 +0000 UTC m=+1643.233955334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls") pod "maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" (UID: "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf") : secret "maas-api-e2e-ait-cb4ef1f3-serving-cert" not found Jul 29 07:42:12.963021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.963000 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-openshift-service-ca\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:12.988335 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:12.988303 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2vct\" (UniqueName: \"kubernetes.io/projected/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-kube-api-access-s2vct\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:13.077658 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.077561 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:13.199212 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.199172 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:13.466469 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.466434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:13.469117 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.469092 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls\") pod \"maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:13.628675 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.628639 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:13.637616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.637578 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:13.755656 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.755624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:13.809504 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.809475 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k"] Jul 29 07:42:13.811408 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:42:13.811380 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c96a1d0_c7e2_4d5f_b57b_522c0d8824bf.slice/crio-fe93c7f57d42a4a605f093d35720c37a2c93d472fa3e83c5e9902ce4e5622ad7 WatchSource:0}: Error finding container fe93c7f57d42a4a605f093d35720c37a2c93d472fa3e83c5e9902ce4e5622ad7: Status 404 returned error can't find the container with id fe93c7f57d42a4a605f093d35720c37a2c93d472fa3e83c5e9902ce4e5622ad7 Jul 29 07:42:13.812780 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:13.812761 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:42:14.657331 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:14.657272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" event={"ID":"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf","Type":"ContainerStarted","Data":"581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae"} Jul 29 07:42:14.657795 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:14.657354 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" event={"ID":"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf","Type":"ContainerStarted","Data":"fe93c7f57d42a4a605f093d35720c37a2c93d472fa3e83c5e9902ce4e5622ad7"} Jul 29 07:42:14.657795 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:14.657383 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:14.696950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:14.696887 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" podStartSLOduration=2.28183178 podStartE2EDuration="2.696868159s" totalImagesPullingTime="415.036379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:14.694705833 +0000 UTC m=+1644.466156632" watchObservedRunningTime="2026-07-29 07:42:14.696868159 +0000 UTC m=+1644.468318958" Jul 29 07:42:20.669408 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:20.669379 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:38.134703 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.134667 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf"] Jul 29 07:42:38.137334 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.137316 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:38.144710 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.144682 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:42:38.152614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.152581 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf"] Jul 29 07:42:38.201106 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.201021 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebb9f80-e346-41c9-8e46-c08220c9a7cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf\" (UID: \"cebb9f80-e346-41c9-8e46-c08220c9a7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:38.302024 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.301990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebb9f80-e346-41c9-8e46-c08220c9a7cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf\" (UID: \"cebb9f80-e346-41c9-8e46-c08220c9a7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:38.302648 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.302624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebb9f80-e346-41c9-8e46-c08220c9a7cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf\" (UID: \"cebb9f80-e346-41c9-8e46-c08220c9a7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:38.447789 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.447744 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:38.599976 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.599942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf"] Jul 29 07:42:38.603562 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:42:38.603532 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebb9f80_e346_41c9_8e46_c08220c9a7cc.slice/crio-b385ab0b6a70194ed4f9db50267b07d31783a0c82e702a82024a2ab4cf5bd47e WatchSource:0}: Error finding container b385ab0b6a70194ed4f9db50267b07d31783a0c82e702a82024a2ab4cf5bd47e: Status 404 returned error can't find the container with id b385ab0b6a70194ed4f9db50267b07d31783a0c82e702a82024a2ab4cf5bd47e Jul 29 07:42:38.786583 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:38.786547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" event={"ID":"cebb9f80-e346-41c9-8e46-c08220c9a7cc","Type":"ContainerStarted","Data":"b385ab0b6a70194ed4f9db50267b07d31783a0c82e702a82024a2ab4cf5bd47e"} Jul 29 07:42:39.798431 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:39.798395 2568 generic.go:356] "Generic (PLEG): container finished" podID="cebb9f80-e346-41c9-8e46-c08220c9a7cc" containerID="f10ac0e8c10afb791fc69d5beb684183f37491f6096470dabfa7f75b82758ca1" exitCode=0 Jul 29 07:42:39.798905 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:39.798482 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" event={"ID":"cebb9f80-e346-41c9-8e46-c08220c9a7cc","Type":"ContainerDied","Data":"f10ac0e8c10afb791fc69d5beb684183f37491f6096470dabfa7f75b82758ca1"} Jul 29 07:42:40.946031 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:40.945994 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:41.030880 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:41.030842 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebb9f80-e346-41c9-8e46-c08220c9a7cc-maas-api-service-ca\") pod \"cebb9f80-e346-41c9-8e46-c08220c9a7cc\" (UID: \"cebb9f80-e346-41c9-8e46-c08220c9a7cc\") " Jul 29 07:42:41.031212 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:41.031189 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebb9f80-e346-41c9-8e46-c08220c9a7cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cebb9f80-e346-41c9-8e46-c08220c9a7cc" (UID: "cebb9f80-e346-41c9-8e46-c08220c9a7cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:42:41.131444 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:41.131358 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebb9f80-e346-41c9-8e46-c08220c9a7cc-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:42:41.811189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:41.811145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" event={"ID":"cebb9f80-e346-41c9-8e46-c08220c9a7cc","Type":"ContainerDied","Data":"b385ab0b6a70194ed4f9db50267b07d31783a0c82e702a82024a2ab4cf5bd47e"} Jul 29 07:42:41.811381 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:41.811204 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b385ab0b6a70194ed4f9db50267b07d31783a0c82e702a82024a2ab4cf5bd47e" Jul 29 07:42:41.811381 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:41.811168 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf" Jul 29 07:42:43.871585 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:43.871546 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k"] Jul 29 07:42:43.872140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:43.871797 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" podUID="2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" containerName="maas-api" containerID="cri-o://581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae" gracePeriod=30 Jul 29 07:42:43.906137 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:43.906103 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:44.139305 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.139221 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:44.220679 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.220597 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:44.262432 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.262390 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-openshift-service-ca\") pod \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " Jul 29 07:42:44.262619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.262513 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls\") pod \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " Jul 29 07:42:44.262619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.262552 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2vct\" (UniqueName: \"kubernetes.io/projected/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-kube-api-access-s2vct\") pod \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\" (UID: \"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf\") " Jul 29 07:42:44.262815 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.262773 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" (UID: "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:42:44.264980 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.264948 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" (UID: "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:42:44.264980 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.264968 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-kube-api-access-s2vct" (OuterVolumeSpecName: "kube-api-access-s2vct") pod "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" (UID: "2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf"). InnerVolumeSpecName "kube-api-access-s2vct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:42:44.364211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.364167 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-maas-api-tls\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.364211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.364202 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2vct\" (UniqueName: \"kubernetes.io/projected/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-kube-api-access-s2vct\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.364211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.364215 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf-openshift-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.396738 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.396646 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf"] Jul 29 07:42:44.398601 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.398564 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cb4ef1f3-4jpcf"] Jul 29 07:42:44.500823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.500793 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:44.699272 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.699237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:44.740370 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.740328 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebb9f80-e346-41c9-8e46-c08220c9a7cc" path="/var/lib/kubelet/pods/cebb9f80-e346-41c9-8e46-c08220c9a7cc/volumes" Jul 29 07:42:44.817636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.817597 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:44.831530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.831494 2568 generic.go:356] "Generic (PLEG): container finished" podID="2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" containerID="581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae" exitCode=0 Jul 29 07:42:44.831721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.831545 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" Jul 29 07:42:44.831721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.831571 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" event={"ID":"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf","Type":"ContainerDied","Data":"581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae"} Jul 29 07:42:44.831721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.831625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k" event={"ID":"2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf","Type":"ContainerDied","Data":"fe93c7f57d42a4a605f093d35720c37a2c93d472fa3e83c5e9902ce4e5622ad7"} Jul 29 07:42:44.831721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.831649 2568 scope.go:117] "RemoveContainer" containerID="581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae" Jul 29 07:42:44.841480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.841461 2568 scope.go:117] "RemoveContainer" containerID="581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae" Jul 29 07:42:44.841812 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:42:44.841788 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae\": container with ID starting with 581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae not found: ID does not exist" containerID="581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae" Jul 29 07:42:44.841879 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.841823 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae"} err="failed to get container status \"581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae\": rpc error: code = NotFound desc = could not find container \"581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae\": container with ID starting with 581d5ef40cd501e9db4eb97b03a847d6adc5500037f7922fafc6e0f6e65aa8ae not found: ID does not exist" Jul 29 07:42:44.851301 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.851266 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k"] Jul 29 07:42:44.855965 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:44.855932 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb4ef1f3-85dd8bd946-42g6k"] Jul 29 07:42:45.066555 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.066465 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:45.163499 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.163463 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:45.301379 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.301344 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:45.778513 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778472 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br"] Jul 29 07:42:45.778895 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778881 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" containerName="maas-api" Jul 29 07:42:45.778946 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778898 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" containerName="maas-api" Jul 29 07:42:45.778946 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778911 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebb9f80-e346-41c9-8e46-c08220c9a7cc" containerName="revoke-keys" Jul 29 07:42:45.778946 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778917 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebb9f80-e346-41c9-8e46-c08220c9a7cc" containerName="revoke-keys" Jul 29 07:42:45.779033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778984 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" containerName="maas-api" Jul 29 07:42:45.779033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.778996 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebb9f80-e346-41c9-8e46-c08220c9a7cc" containerName="revoke-keys" Jul 29 07:42:45.783478 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.783452 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.789373 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.789343 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:42:45.790847 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.790816 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8672bcd3-gw-openshift-default-dockercfg-gnnp4\"" Jul 29 07:42:45.822448 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.822395 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br"] Jul 29 07:42:45.878156 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-credential-socket\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878385 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-certs\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878385 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-podinfo\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878385 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-envoy\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878385 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878349 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-data\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878776 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-socket\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878776 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-token\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878776 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878508 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kqv\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-kube-api-access-k4kqv\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.878776 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.878618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istiod-ca-cert\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.901606 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.901431 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:45.979533 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-credential-socket\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979533 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-certs\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-podinfo\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-envoy\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-data\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-socket\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-token\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kqv\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-kube-api-access-k4kqv\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.979811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istiod-ca-cert\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.980239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.979991 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-credential-socket\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.980239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.980001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-certs\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.980239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.980079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-socket\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.980239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.980176 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-data\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.980565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.980544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istiod-ca-cert\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.982277 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.982254 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-envoy\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.982439 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.982420 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-podinfo\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.995515 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.995410 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kqv\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-kube-api-access-k4kqv\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:45.997395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:45.997367 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-token\") pod \"e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:46.035735 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:46.035617 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:46.094614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:46.094580 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:46.270636 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:42:46.270587 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c01b9a_c7dd_478e_84f4_6c7628e986bb.slice/crio-915c40a97fc7210013eae3049aa4c0bd707272c92147930cc8db968eb7015856 WatchSource:0}: Error finding container 915c40a97fc7210013eae3049aa4c0bd707272c92147930cc8db968eb7015856: Status 404 returned error can't find the container with id 915c40a97fc7210013eae3049aa4c0bd707272c92147930cc8db968eb7015856 Jul 29 07:42:46.270821 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:46.270788 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br"] Jul 29 07:42:46.738827 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:46.738787 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf" path="/var/lib/kubelet/pods/2c96a1d0-c7e2-4d5f-b57b-522c0d8824bf/volumes" Jul 29 07:42:46.846639 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:46.846600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" event={"ID":"45c01b9a-c7dd-478e-84f4-6c7628e986bb","Type":"ContainerStarted","Data":"915c40a97fc7210013eae3049aa4c0bd707272c92147930cc8db968eb7015856"} Jul 29 07:42:48.888564 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:48.888528 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:42:48.888849 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:48.888641 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:42:48.888849 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:48.888672 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:42:49.865150 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:49.865113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" event={"ID":"45c01b9a-c7dd-478e-84f4-6c7628e986bb","Type":"ContainerStarted","Data":"3201286feff7c370021b2532f73178f753ea448ca0a39b3901ba385c97450790"} Jul 29 07:42:49.897139 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:49.897078 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podStartSLOduration=2.281692176 podStartE2EDuration="4.897057207s" totalImagesPullingTime="2.615365031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:49.894507749 +0000 UTC m=+1679.665958549" watchObservedRunningTime="2026-07-29 07:42:49.897057207 +0000 UTC m=+1679.668508000" Jul 29 07:42:50.094907 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:50.094870 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:50.100570 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:50.100539 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:50.871354 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:50.871315 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:50.872458 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:50.872434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:42:51.488367 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.488323 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:51.584850 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.584817 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755"] Jul 29 07:42:51.588254 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.588228 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.591239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.591214 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8672bcd3-serving-cert\"" Jul 29 07:42:51.604613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.604578 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755"] Jul 29 07:42:51.627744 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.627682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:51.739496 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.739379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-openshift-service-ca\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.739496 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.739440 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.739730 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.739516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt49j\" (UniqueName: \"kubernetes.io/projected/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-kube-api-access-pt49j\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.840324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.840277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-openshift-service-ca\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.840515 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.840351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.840515 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.840379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt49j\" (UniqueName: \"kubernetes.io/projected/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-kube-api-access-pt49j\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.840656 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:42:51.840514 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-serving-cert: secret "maas-api-e2e-migrate-8672bcd3-serving-cert" not found Jul 29 07:42:51.840656 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:42:51.840610 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls podName:3f310d1e-4daf-4817-8ffb-f4c4ebe2769e nodeName:}" failed. No retries permitted until 2026-07-29 07:42:52.340587707 +0000 UTC m=+1682.112038486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls") pod "maas-api-e2e-migrate-8672bcd3-54999859b-v8755" (UID: "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e") : secret "maas-api-e2e-migrate-8672bcd3-serving-cert" not found Jul 29 07:42:51.841195 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.841174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-openshift-service-ca\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.853743 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.853705 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt49j\" (UniqueName: \"kubernetes.io/projected/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-kube-api-access-pt49j\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:51.955640 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:51.955595 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:52.211292 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.211255 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:52.347171 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.347132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:52.349871 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.349835 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls\") pod \"maas-api-e2e-migrate-8672bcd3-54999859b-v8755\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:52.365972 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.365939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:42:52.499899 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.499795 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:52.677784 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.677756 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755"] Jul 29 07:42:52.679761 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:42:52.679731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f310d1e_4daf_4817_8ffb_f4c4ebe2769e.slice/crio-62b7c0b02186069380aea36d867954f9af38f3d376289a0920d6ede519dfc36b WatchSource:0}: Error finding container 62b7c0b02186069380aea36d867954f9af38f3d376289a0920d6ede519dfc36b: Status 404 returned error can't find the container with id 62b7c0b02186069380aea36d867954f9af38f3d376289a0920d6ede519dfc36b Jul 29 07:42:52.883348 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:52.883247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" event={"ID":"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e","Type":"ContainerStarted","Data":"62b7c0b02186069380aea36d867954f9af38f3d376289a0920d6ede519dfc36b"} Jul 29 07:42:53.890534 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:53.890493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" event={"ID":"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e","Type":"ContainerStarted","Data":"a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73"} Jul 29 07:42:53.890953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:53.890622 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:42:53.918514 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:53.918457 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" podStartSLOduration=2.543998433 podStartE2EDuration="2.918440452s" totalImagesPullingTime="374.442019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:53.917399294 +0000 UTC m=+1683.688850096" watchObservedRunningTime="2026-07-29 07:42:53.918440452 +0000 UTC m=+1683.689891284" Jul 29 07:42:59.902168 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:42:59.902135 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:43:03.964969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:03.964934 2568 generic.go:356] "Generic (PLEG): container finished" podID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerID="3201286feff7c370021b2532f73178f753ea448ca0a39b3901ba385c97450790" exitCode=0 Jul 29 07:43:03.965450 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:03.965011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" event={"ID":"45c01b9a-c7dd-478e-84f4-6c7628e986bb","Type":"ContainerDied","Data":"3201286feff7c370021b2532f73178f753ea448ca0a39b3901ba385c97450790"} Jul 29 07:43:03.965450 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:03.965424 2568 scope.go:117] "RemoveContainer" containerID="3201286feff7c370021b2532f73178f753ea448ca0a39b3901ba385c97450790" Jul 29 07:43:03.966668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:03.966631 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:03.966768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:03.966715 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:03.966768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:03.966754 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:04.973077 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:04.973021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" event={"ID":"45c01b9a-c7dd-478e-84f4-6c7628e986bb","Type":"ContainerStarted","Data":"fd5088783056dda280aded4d0c17710280ed668e79234b04b0378b950af31d7b"} Jul 29 07:43:05.095767 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:05.095725 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:43:05.097114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:05.097084 2568 patch_prober.go:28] interesting pod/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 29 07:43:05.097204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:05.097148 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 29 07:43:05.871499 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:05.871457 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:43:06.095316 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:06.095279 2568 patch_prober.go:28] interesting pod/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 29 07:43:06.095736 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:06.095344 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 29 07:43:07.095617 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:07.095573 2568 patch_prober.go:28] interesting pod/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 29 07:43:07.096076 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:07.095640 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 29 07:43:08.095944 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:08.095910 2568 patch_prober.go:28] interesting pod/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 29 07:43:08.096391 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:08.095966 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 29 07:43:09.099523 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:09.099491 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:43:09.100551 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:09.100531 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:43:16.657545 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.657448 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89"] Jul 29 07:43:16.664696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.664668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:16.668215 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.668194 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:43:16.676265 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.676235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bed7d23-dbf2-4e96-86df-0895f7c42d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89\" (UID: \"2bed7d23-dbf2-4e96-86df-0895f7c42d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:16.686668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.686631 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89"] Jul 29 07:43:16.777695 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.777647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bed7d23-dbf2-4e96-86df-0895f7c42d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89\" (UID: \"2bed7d23-dbf2-4e96-86df-0895f7c42d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:16.778364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.778342 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bed7d23-dbf2-4e96-86df-0895f7c42d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89\" (UID: \"2bed7d23-dbf2-4e96-86df-0895f7c42d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:16.975697 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:16.975561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:17.153916 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:17.153885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89"] Jul 29 07:43:17.156897 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:43:17.156863 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bed7d23_dbf2_4e96_86df_0895f7c42d80.slice/crio-f7acff84b5747b60ea2cb1617a99d7516c3b6ae6f8c39e62a16577dfd8fa7529 WatchSource:0}: Error finding container f7acff84b5747b60ea2cb1617a99d7516c3b6ae6f8c39e62a16577dfd8fa7529: Status 404 returned error can't find the container with id f7acff84b5747b60ea2cb1617a99d7516c3b6ae6f8c39e62a16577dfd8fa7529 Jul 29 07:43:18.046398 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:18.046362 2568 generic.go:356] "Generic (PLEG): container finished" podID="2bed7d23-dbf2-4e96-86df-0895f7c42d80" containerID="248288df7ba7b1dd4d85615bf9c9bbd3dcb6fad47bb68a2425c135df5b0f9c9f" exitCode=0 Jul 29 07:43:18.046823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:18.046445 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" event={"ID":"2bed7d23-dbf2-4e96-86df-0895f7c42d80","Type":"ContainerDied","Data":"248288df7ba7b1dd4d85615bf9c9bbd3dcb6fad47bb68a2425c135df5b0f9c9f"} Jul 29 07:43:18.046823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:18.046499 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" event={"ID":"2bed7d23-dbf2-4e96-86df-0895f7c42d80","Type":"ContainerStarted","Data":"f7acff84b5747b60ea2cb1617a99d7516c3b6ae6f8c39e62a16577dfd8fa7529"} Jul 29 07:43:19.193554 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:19.193523 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:19.300894 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:19.300852 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bed7d23-dbf2-4e96-86df-0895f7c42d80-maas-api-service-ca\") pod \"2bed7d23-dbf2-4e96-86df-0895f7c42d80\" (UID: \"2bed7d23-dbf2-4e96-86df-0895f7c42d80\") " Jul 29 07:43:19.301261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:19.301239 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bed7d23-dbf2-4e96-86df-0895f7c42d80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bed7d23-dbf2-4e96-86df-0895f7c42d80" (UID: "2bed7d23-dbf2-4e96-86df-0895f7c42d80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:19.402440 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:19.402408 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bed7d23-dbf2-4e96-86df-0895f7c42d80-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:20.058312 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:20.058280 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" Jul 29 07:43:20.058312 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:20.058301 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89" event={"ID":"2bed7d23-dbf2-4e96-86df-0895f7c42d80","Type":"ContainerDied","Data":"f7acff84b5747b60ea2cb1617a99d7516c3b6ae6f8c39e62a16577dfd8fa7529"} Jul 29 07:43:20.058555 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:20.058349 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7acff84b5747b60ea2cb1617a99d7516c3b6ae6f8c39e62a16577dfd8fa7529" Jul 29 07:43:22.100423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.100383 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755"] Jul 29 07:43:22.100828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.100596 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" podUID="3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" containerName="maas-api" containerID="cri-o://a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73" gracePeriod=30 Jul 29 07:43:22.233316 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.233249 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:22.378170 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.378138 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:43:22.431686 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.431638 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls\") pod \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " Jul 29 07:43:22.431903 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.431848 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-openshift-service-ca\") pod \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " Jul 29 07:43:22.431903 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.431881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt49j\" (UniqueName: \"kubernetes.io/projected/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-kube-api-access-pt49j\") pod \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\" (UID: \"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e\") " Jul 29 07:43:22.432303 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.432275 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" (UID: "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:22.434133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.434105 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" (UID: "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:43:22.434280 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.434220 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-kube-api-access-pt49j" (OuterVolumeSpecName: "kube-api-access-pt49j") pod "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" (UID: "3f310d1e-4daf-4817-8ffb-f4c4ebe2769e"). InnerVolumeSpecName "kube-api-access-pt49j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:22.435388 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.435349 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:22.533364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.533314 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-openshift-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:22.533364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.533353 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt49j\" (UniqueName: \"kubernetes.io/projected/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-kube-api-access-pt49j\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:22.533364 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.533368 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e-maas-api-tls\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:22.759663 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.759600 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:22.779412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.779377 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89"] Jul 29 07:43:22.794053 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.794001 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8672bcd3-95d89"] Jul 29 07:43:22.956791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.956747 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br"] Jul 29 07:43:22.956986 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:22.956965 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" containerID="cri-o://fd5088783056dda280aded4d0c17710280ed668e79234b04b0378b950af31d7b" gracePeriod=30 Jul 29 07:43:23.007428 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.007332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:23.078460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.078427 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" containerID="a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73" exitCode=0 Jul 29 07:43:23.078629 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.078473 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" event={"ID":"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e","Type":"ContainerDied","Data":"a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73"} Jul 29 07:43:23.078629 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.078517 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" event={"ID":"3f310d1e-4daf-4817-8ffb-f4c4ebe2769e","Type":"ContainerDied","Data":"62b7c0b02186069380aea36d867954f9af38f3d376289a0920d6ede519dfc36b"} Jul 29 07:43:23.078629 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.078539 2568 scope.go:117] "RemoveContainer" containerID="a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73" Jul 29 07:43:23.078629 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.078474 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755" Jul 29 07:43:23.088704 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.088683 2568 scope.go:117] "RemoveContainer" containerID="a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73" Jul 29 07:43:23.089068 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:43:23.089028 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73\": container with ID starting with a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73 not found: ID does not exist" containerID="a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73" Jul 29 07:43:23.089140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.089084 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73"} err="failed to get container status \"a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73\": rpc error: code = NotFound desc = could not find container \"a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73\": container with ID starting with a1c7800fb162acf07adf311196b2d7b3f8caefe5a82dc1efe8a5521695c0fa73 not found: ID does not exist" Jul 29 07:43:23.122054 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.122008 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755"] Jul 29 07:43:23.128711 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.128674 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8672bcd3-54999859b-v8755"] Jul 29 07:43:23.227324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.227289 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:23.576831 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:23.576796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:24.046232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.046191 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5"] Jul 29 07:43:24.046687 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.046671 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" containerName="maas-api" Jul 29 07:43:24.046739 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.046689 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" containerName="maas-api" Jul 29 07:43:24.046739 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.046699 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bed7d23-dbf2-4e96-86df-0895f7c42d80" containerName="revoke-keys" Jul 29 07:43:24.046739 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.046704 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bed7d23-dbf2-4e96-86df-0895f7c42d80" containerName="revoke-keys" Jul 29 07:43:24.051343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.047198 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bed7d23-dbf2-4e96-86df-0895f7c42d80" containerName="revoke-keys" Jul 29 07:43:24.051343 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.047292 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" containerName="maas-api" Jul 29 07:43:24.054483 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.054446 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.066649 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.066621 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d04dd228-openshift-default-dockercfg-xn8qw\"" Jul 29 07:43:24.102823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.102780 2568 patch_prober.go:28] interesting pod/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 07:43:24.103013 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.102862 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:43:24.125620 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.125581 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5"] Jul 29 07:43:24.147663 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147623 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-certs\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.147893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147682 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87p92\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-kube-api-access-87p92\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.147893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-istio-token\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.147893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147760 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-credential-socket\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.147893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147789 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-socket\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.147893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d443172f-397e-4e3c-9558-8e2b54b78726-istio-podinfo\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.148177 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.147917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d443172f-397e-4e3c-9558-8e2b54b78726-istiod-ca-cert\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.148177 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.148004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-envoy\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.148177 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.148032 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-data\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.174366 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.174332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:24.249616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249563 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-envoy\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-data\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-certs\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87p92\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-kube-api-access-87p92\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-istio-token\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-credential-socket\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-socket\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.249914 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d443172f-397e-4e3c-9558-8e2b54b78726-istio-podinfo\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.250287 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.249920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d443172f-397e-4e3c-9558-8e2b54b78726-istiod-ca-cert\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.250287 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.250190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-certs\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.250392 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.250369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-socket\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.250452 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.250433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-credential-socket\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.250551 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.250535 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-data\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.250748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.250729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d443172f-397e-4e3c-9558-8e2b54b78726-istiod-ca-cert\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.252261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.252234 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-envoy\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.252908 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.252886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d443172f-397e-4e3c-9558-8e2b54b78726-istio-podinfo\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.280582 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.280548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-istio-token\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.282486 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.282450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87p92\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-kube-api-access-87p92\") pod \"e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.365657 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.365563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:24.365944 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.365773 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:24.568378 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.568338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5"] Jul 29 07:43:24.574403 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:43:24.574373 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd443172f_397e_4e3c_9558_8e2b54b78726.slice/crio-cbfced4b6ca8a49c7201e4eb4099ffa458a5b1563e6ec809c418608c172928a2 WatchSource:0}: Error finding container cbfced4b6ca8a49c7201e4eb4099ffa458a5b1563e6ec809c418608c172928a2: Status 404 returned error can't find the container with id cbfced4b6ca8a49c7201e4eb4099ffa458a5b1563e6ec809c418608c172928a2 Jul 29 07:43:24.576593 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.576563 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:24.576668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.576631 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:24.576668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.576662 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:24.740965 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.740925 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bed7d23-dbf2-4e96-86df-0895f7c42d80" path="/var/lib/kubelet/pods/2bed7d23-dbf2-4e96-86df-0895f7c42d80/volumes" Jul 29 07:43:24.741484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:24.741462 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f310d1e-4daf-4817-8ffb-f4c4ebe2769e" path="/var/lib/kubelet/pods/3f310d1e-4daf-4817-8ffb-f4c4ebe2769e/volumes" Jul 29 07:43:25.092411 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:25.092372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" event={"ID":"d443172f-397e-4e3c-9558-8e2b54b78726","Type":"ContainerStarted","Data":"d9f7c9d7d090d07ff1d06de0947841bbb88f4cbcbacc970367afb8c2c6b3db22"} Jul 29 07:43:25.092612 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:25.092424 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" event={"ID":"d443172f-397e-4e3c-9558-8e2b54b78726","Type":"ContainerStarted","Data":"cbfced4b6ca8a49c7201e4eb4099ffa458a5b1563e6ec809c418608c172928a2"} Jul 29 07:43:25.138170 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:25.138110 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podStartSLOduration=2.138089302 podStartE2EDuration="2.138089302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:43:25.136268531 +0000 UTC m=+1714.907719330" watchObservedRunningTime="2026-07-29 07:43:25.138089302 +0000 UTC m=+1714.909540092" Jul 29 07:43:25.366133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:25.366008 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:25.371432 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:25.371403 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:26.097956 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:26.097923 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:26.099034 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:26.099009 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:26.659565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:26.659530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:26.957336 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:26.957224 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:27.243291 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:27.243200 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:28.113862 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.113824 2568 generic.go:356] "Generic (PLEG): container finished" podID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerID="fd5088783056dda280aded4d0c17710280ed668e79234b04b0378b950af31d7b" exitCode=0 Jul 29 07:43:28.114374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.113883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" event={"ID":"45c01b9a-c7dd-478e-84f4-6c7628e986bb","Type":"ContainerDied","Data":"fd5088783056dda280aded4d0c17710280ed668e79234b04b0378b950af31d7b"} Jul 29 07:43:28.114374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.113965 2568 scope.go:117] "RemoveContainer" containerID="3201286feff7c370021b2532f73178f753ea448ca0a39b3901ba385c97450790" Jul 29 07:43:28.247751 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.247715 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:43:28.295563 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295531 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-token\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.295772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295570 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-podinfo\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.295772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295599 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-socket\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.295772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295627 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-data\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.295772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295693 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4kqv\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-kube-api-access-k4kqv\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.295772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295764 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-credential-socket\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.296069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295781 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istiod-ca-cert\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.296069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295815 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-certs\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.296069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295844 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-envoy\") pod \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\" (UID: \"45c01b9a-c7dd-478e-84f4-6c7628e986bb\") " Jul 29 07:43:28.296069 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.295959 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:43:28.296248 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.296166 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.296408 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.296375 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:43:28.296586 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.296489 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:43:28.296940 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.296913 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:43:28.297023 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.296984 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:28.298568 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.298544 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:43:28.298668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.298571 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-kube-api-access-k4kqv" (OuterVolumeSpecName: "kube-api-access-k4kqv") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "kube-api-access-k4kqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:28.298668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.298656 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:28.298786 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.298673 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45c01b9a-c7dd-478e-84f4-6c7628e986bb" (UID: "45c01b9a-c7dd-478e-84f4-6c7628e986bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:43:28.301462 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.301431 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:28.397480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397422 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397472 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397485 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4kqv\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-kube-api-access-k4kqv\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397757 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397497 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397757 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397511 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397757 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397522 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397757 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397536 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.397757 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.397547 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01b9a-c7dd-478e-84f4-6c7628e986bb-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:28.497110 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:28.496976 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:29.127828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:29.127745 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" Jul 29 07:43:29.127828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:29.127767 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br" event={"ID":"45c01b9a-c7dd-478e-84f4-6c7628e986bb","Type":"ContainerDied","Data":"915c40a97fc7210013eae3049aa4c0bd707272c92147930cc8db968eb7015856"} Jul 29 07:43:29.128353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:29.127851 2568 scope.go:117] "RemoveContainer" containerID="fd5088783056dda280aded4d0c17710280ed668e79234b04b0378b950af31d7b" Jul 29 07:43:29.162157 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:29.162112 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br"] Jul 29 07:43:29.167980 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:29.167943 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8672bcd3-gw-openshift-default-6985945d6d-sx6br"] Jul 29 07:43:30.738370 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:30.738340 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" path="/var/lib/kubelet/pods/45c01b9a-c7dd-478e-84f4-6c7628e986bb/volumes" Jul 29 07:43:39.200636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:39.200599 2568 generic.go:356] "Generic (PLEG): container finished" podID="d443172f-397e-4e3c-9558-8e2b54b78726" containerID="d9f7c9d7d090d07ff1d06de0947841bbb88f4cbcbacc970367afb8c2c6b3db22" exitCode=0 Jul 29 07:43:39.200636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:39.200643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" event={"ID":"d443172f-397e-4e3c-9558-8e2b54b78726","Type":"ContainerDied","Data":"d9f7c9d7d090d07ff1d06de0947841bbb88f4cbcbacc970367afb8c2c6b3db22"} Jul 29 07:43:39.201258 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:39.201099 2568 scope.go:117] "RemoveContainer" containerID="d9f7c9d7d090d07ff1d06de0947841bbb88f4cbcbacc970367afb8c2c6b3db22" Jul 29 07:43:39.202206 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:39.202173 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:39.202280 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:39.202249 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:39.202280 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:39.202278 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:43:40.209233 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:40.209196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" event={"ID":"d443172f-397e-4e3c-9558-8e2b54b78726","Type":"ContainerStarted","Data":"3b740beeb087299419ab3f7527b2e54fc4bf89211283ef5cb598759a5e5df4ce"} Jul 29 07:43:40.366391 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:40.366349 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:40.367734 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:40.367704 2568 patch_prober.go:28] interesting pod/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 07:43:40.367883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:40.367761 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 07:43:41.098659 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:41.098604 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:41.366302 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:41.366207 2568 patch_prober.go:28] interesting pod/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 07:43:41.366302 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:41.366284 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 07:43:42.366096 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:42.366055 2568 patch_prober.go:28] interesting pod/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 07:43:42.366318 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:42.366122 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 07:43:43.366006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:43.365961 2568 patch_prober.go:28] interesting pod/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 07:43:43.366242 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:43.366024 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 07:43:44.370556 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:44.370524 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:44.371522 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:44.371504 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:43:52.418249 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.418197 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk"] Jul 29 07:43:52.418702 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.418612 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" Jul 29 07:43:52.418702 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.418624 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" Jul 29 07:43:52.418803 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.418706 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" Jul 29 07:43:52.418803 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.418717 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" Jul 29 07:43:52.423392 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.423369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:52.427184 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.427160 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:43:52.473031 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.472994 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk"] Jul 29 07:43:52.531204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.531169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985a14f-0421-4f67-80b1-fa9f8aea859f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk\" (UID: \"2985a14f-0421-4f67-80b1-fa9f8aea859f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:52.632428 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.632388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985a14f-0421-4f67-80b1-fa9f8aea859f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk\" (UID: \"2985a14f-0421-4f67-80b1-fa9f8aea859f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:52.633176 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.633156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985a14f-0421-4f67-80b1-fa9f8aea859f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk\" (UID: \"2985a14f-0421-4f67-80b1-fa9f8aea859f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:52.734217 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.734112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:52.887437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:52.887398 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk"] Jul 29 07:43:52.890762 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:43:52.890731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2985a14f_0421_4f67_80b1_fa9f8aea859f.slice/crio-e30dd4aa3027b731fab9b38649c37f5ca46bd8d2aa47e2f2c11af35903c3a45e WatchSource:0}: Error finding container e30dd4aa3027b731fab9b38649c37f5ca46bd8d2aa47e2f2c11af35903c3a45e: Status 404 returned error can't find the container with id e30dd4aa3027b731fab9b38649c37f5ca46bd8d2aa47e2f2c11af35903c3a45e Jul 29 07:43:53.286439 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:53.286401 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" event={"ID":"2985a14f-0421-4f67-80b1-fa9f8aea859f","Type":"ContainerStarted","Data":"366dc28a3967ac9dd98ba61735aee8b23e966c3ec47661a9f39a5ab0040644ac"} Jul 29 07:43:53.286618 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:53.286456 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" event={"ID":"2985a14f-0421-4f67-80b1-fa9f8aea859f","Type":"ContainerStarted","Data":"e30dd4aa3027b731fab9b38649c37f5ca46bd8d2aa47e2f2c11af35903c3a45e"} Jul 29 07:43:53.309723 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:53.309665 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" podStartSLOduration=1.309647437 podStartE2EDuration="1.309647437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:43:53.307654657 +0000 UTC m=+1743.079105461" watchObservedRunningTime="2026-07-29 07:43:53.309647437 +0000 UTC m=+1743.081098235" Jul 29 07:43:54.293543 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:54.293504 2568 generic.go:356] "Generic (PLEG): container finished" podID="2985a14f-0421-4f67-80b1-fa9f8aea859f" containerID="366dc28a3967ac9dd98ba61735aee8b23e966c3ec47661a9f39a5ab0040644ac" exitCode=0 Jul 29 07:43:54.293951 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:54.293568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" event={"ID":"2985a14f-0421-4f67-80b1-fa9f8aea859f","Type":"ContainerDied","Data":"366dc28a3967ac9dd98ba61735aee8b23e966c3ec47661a9f39a5ab0040644ac"} Jul 29 07:43:55.445379 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:55.445337 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:55.458089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:55.458028 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985a14f-0421-4f67-80b1-fa9f8aea859f-maas-api-service-ca\") pod \"2985a14f-0421-4f67-80b1-fa9f8aea859f\" (UID: \"2985a14f-0421-4f67-80b1-fa9f8aea859f\") " Jul 29 07:43:55.458412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:55.458389 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2985a14f-0421-4f67-80b1-fa9f8aea859f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2985a14f-0421-4f67-80b1-fa9f8aea859f" (UID: "2985a14f-0421-4f67-80b1-fa9f8aea859f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:55.558756 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:55.558654 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985a14f-0421-4f67-80b1-fa9f8aea859f-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:43:56.305500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:56.305469 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" Jul 29 07:43:56.305690 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:56.305500 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk" event={"ID":"2985a14f-0421-4f67-80b1-fa9f8aea859f","Type":"ContainerDied","Data":"e30dd4aa3027b731fab9b38649c37f5ca46bd8d2aa47e2f2c11af35903c3a45e"} Jul 29 07:43:56.305690 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:56.305547 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30dd4aa3027b731fab9b38649c37f5ca46bd8d2aa47e2f2c11af35903c3a45e" Jul 29 07:43:57.507352 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:57.507292 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:57.686510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:57.686472 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:57.707537 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:57.707497 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk"] Jul 29 07:43:57.710933 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:57.710902 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d04dd228-nkvsk"] Jul 29 07:43:57.754528 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:57.754489 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:58.738423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:58.738388 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2985a14f-0421-4f67-80b1-fa9f8aea859f" path="/var/lib/kubelet/pods/2985a14f-0421-4f67-80b1-fa9f8aea859f/volumes" Jul 29 07:43:59.092485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.092383 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5"] Jul 29 07:43:59.092683 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.092646 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" containerID="cri-o://3b740beeb087299419ab3f7527b2e54fc4bf89211283ef5cb598759a5e5df4ce" gracePeriod=30 Jul 29 07:43:59.167232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.167195 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:59.355341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.355219 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:59.373619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.373581 2568 patch_prober.go:28] interesting pod/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:43:59.373792 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.373645 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:43:59.587877 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.587839 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:59.899033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.898988 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn"] Jul 29 07:43:59.899565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.899424 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" Jul 29 07:43:59.899565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.899440 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c01b9a-c7dd-478e-84f4-6c7628e986bb" containerName="istio-proxy" Jul 29 07:43:59.899565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.899459 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985a14f-0421-4f67-80b1-fa9f8aea859f" containerName="revoke-keys" Jul 29 07:43:59.899565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.899465 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985a14f-0421-4f67-80b1-fa9f8aea859f" containerName="revoke-keys" Jul 29 07:43:59.899565 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.899545 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2985a14f-0421-4f67-80b1-fa9f8aea859f" containerName="revoke-keys" Jul 29 07:43:59.903091 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.903065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.906029 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.906005 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3564323f-openshift-default-dockercfg-9c2j8\"" Jul 29 07:43:59.918854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.918824 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn"] Jul 29 07:43:59.930406 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.930371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:43:59.997569 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-socket\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-credential-socket\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-certs\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997650 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-istio-token\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-envoy\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pc5\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-kube-api-access-97pc5\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85812904-c299-4c41-991e-a2158b10abec-istiod-ca-cert\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85812904-c299-4c41-991e-a2158b10abec-istio-podinfo\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:43:59.997953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:43:59.997843 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-data\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.098880 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.098833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85812904-c299-4c41-991e-a2158b10abec-istiod-ca-cert\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.098891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85812904-c299-4c41-991e-a2158b10abec-istio-podinfo\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.098922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-data\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.098943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-socket\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.098994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-credential-socket\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-certs\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-istio-token\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-envoy\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pc5\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-kube-api-access-97pc5\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099532 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099485 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-credential-socket\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099656 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85812904-c299-4c41-991e-a2158b10abec-istiod-ca-cert\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099795 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-socket\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.099896 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.099874 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-data\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.100024 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.100001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-certs\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.101604 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.101582 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-envoy\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.101758 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.101739 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85812904-c299-4c41-991e-a2158b10abec-istio-podinfo\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.109657 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.109616 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-istio-token\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.109809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.109684 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pc5\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-kube-api-access-97pc5\") pod \"e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.215211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.215172 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:00.242570 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.242533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:00.376932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.376898 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn"] Jul 29 07:44:00.378608 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:44:00.378574 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85812904_c299_4c41_991e_a2158b10abec.slice/crio-fb5b15fcbbc2db9fe60002d9d2e59c2f77454fa44ea0b01a1e9204b927597485 WatchSource:0}: Error finding container fb5b15fcbbc2db9fe60002d9d2e59c2f77454fa44ea0b01a1e9204b927597485: Status 404 returned error can't find the container with id fb5b15fcbbc2db9fe60002d9d2e59c2f77454fa44ea0b01a1e9204b927597485 Jul 29 07:44:00.381090 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.381056 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:44:00.381163 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.381130 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:44:00.381163 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:00.381158 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:44:01.336548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:01.336509 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" event={"ID":"85812904-c299-4c41-991e-a2158b10abec","Type":"ContainerStarted","Data":"f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962"} Jul 29 07:44:01.336984 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:01.336569 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" event={"ID":"85812904-c299-4c41-991e-a2158b10abec","Type":"ContainerStarted","Data":"fb5b15fcbbc2db9fe60002d9d2e59c2f77454fa44ea0b01a1e9204b927597485"} Jul 29 07:44:01.362573 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:01.362503 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podStartSLOduration=2.362485552 podStartE2EDuration="2.362485552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:44:01.361014091 +0000 UTC m=+1751.132464890" watchObservedRunningTime="2026-07-29 07:44:01.362485552 +0000 UTC m=+1751.133936351" Jul 29 07:44:02.216276 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:02.216231 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:02.222061 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:02.222004 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:02.342794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:02.342758 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:02.343857 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:02.343836 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:02.749723 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:02.749685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:02.876369 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:02.876333 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:03.170423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:03.170391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:03.514055 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:03.513953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:04.363768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.363671 2568 generic.go:356] "Generic (PLEG): container finished" podID="d443172f-397e-4e3c-9558-8e2b54b78726" containerID="3b740beeb087299419ab3f7527b2e54fc4bf89211283ef5cb598759a5e5df4ce" exitCode=0 Jul 29 07:44:04.363768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.363744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" event={"ID":"d443172f-397e-4e3c-9558-8e2b54b78726","Type":"ContainerDied","Data":"3b740beeb087299419ab3f7527b2e54fc4bf89211283ef5cb598759a5e5df4ce"} Jul 29 07:44:04.364001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.363850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" event={"ID":"d443172f-397e-4e3c-9558-8e2b54b78726","Type":"ContainerDied","Data":"cbfced4b6ca8a49c7201e4eb4099ffa458a5b1563e6ec809c418608c172928a2"} Jul 29 07:44:04.364001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.363876 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfced4b6ca8a49c7201e4eb4099ffa458a5b1563e6ec809c418608c172928a2" Jul 29 07:44:04.364001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.363889 2568 scope.go:117] "RemoveContainer" containerID="d9f7c9d7d090d07ff1d06de0947841bbb88f4cbcbacc970367afb8c2c6b3db22" Jul 29 07:44:04.367276 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.367253 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:44:04.439350 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439308 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d443172f-397e-4e3c-9558-8e2b54b78726-istio-podinfo\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439350 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439355 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-data\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439381 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-certs\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439423 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d443172f-397e-4e3c-9558-8e2b54b78726-istiod-ca-cert\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439450 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87p92\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-kube-api-access-87p92\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439472 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-istio-token\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439495 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-socket\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439543 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-credential-socket\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439565 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-envoy\") pod \"d443172f-397e-4e3c-9558-8e2b54b78726\" (UID: \"d443172f-397e-4e3c-9558-8e2b54b78726\") " Jul 29 07:44:04.439942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439784 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:04.439942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439801 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:04.439942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439877 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d443172f-397e-4e3c-9558-8e2b54b78726-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:44:04.440126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.439991 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.440126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.440011 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d443172f-397e-4e3c-9558-8e2b54b78726-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.440126 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.440076 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.440287 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.440165 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:04.440287 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.440277 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-data" (OuterVolumeSpecName: "istio-data") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:04.442128 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.442100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-kube-api-access-87p92" (OuterVolumeSpecName: "kube-api-access-87p92") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "kube-api-access-87p92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:04.442279 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.442146 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:04.442279 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.442244 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d443172f-397e-4e3c-9558-8e2b54b78726-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:44:04.442404 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.442385 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-istio-token" (OuterVolumeSpecName: "istio-token") pod "d443172f-397e-4e3c-9558-8e2b54b78726" (UID: "d443172f-397e-4e3c-9558-8e2b54b78726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:04.541495 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.541450 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87p92\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-kube-api-access-87p92\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.541888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.541498 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d443172f-397e-4e3c-9558-8e2b54b78726-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.541888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.541533 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.541888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.541556 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.541888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.541566 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d443172f-397e-4e3c-9558-8e2b54b78726-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.541888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.541574 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d443172f-397e-4e3c-9558-8e2b54b78726-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:04.555774 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.555742 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:04.924248 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:04.924213 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:05.370754 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:05.370719 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5" Jul 29 07:44:05.474288 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:05.474248 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5"] Jul 29 07:44:05.482683 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:05.482651 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d04dd228-openshift-default-5f8f6fdb8d-zqfh5"] Jul 29 07:44:06.738668 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:06.738631 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" path="/var/lib/kubelet/pods/d443172f-397e-4e3c-9558-8e2b54b78726/volumes" Jul 29 07:44:16.225349 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:44:16.225289 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85812904_c299_4c41_991e_a2158b10abec.slice/crio-f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-27023-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85812904_c299_4c41_991e_a2158b10abec.slice/crio-conmon-f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962.scope\": RecentStats: unable to find data in memory cache]" Jul 29 07:44:16.435142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:16.435103 2568 generic.go:356] "Generic (PLEG): container finished" podID="85812904-c299-4c41-991e-a2158b10abec" containerID="f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962" exitCode=0 Jul 29 07:44:16.435323 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:16.435186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" event={"ID":"85812904-c299-4c41-991e-a2158b10abec","Type":"ContainerDied","Data":"f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962"} Jul 29 07:44:16.435624 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:16.435608 2568 scope.go:117] "RemoveContainer" containerID="f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962" Jul 29 07:44:16.436850 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:16.436816 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:44:16.436932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:16.436916 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:44:16.436973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:16.436958 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:44:17.343431 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:17.343396 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:17.448080 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:17.447131 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" event={"ID":"85812904-c299-4c41-991e-a2158b10abec","Type":"ContainerStarted","Data":"53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825"} Jul 29 07:44:18.215753 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:18.215708 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:18.216797 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:18.216772 2568 patch_prober.go:28] interesting pod/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 07:44:18.216861 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:18.216843 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 07:44:19.215986 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:19.215947 2568 patch_prober.go:28] interesting pod/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 07:44:19.216450 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:19.216025 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 07:44:20.216264 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:20.216215 2568 patch_prober.go:28] interesting pod/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 07:44:20.216687 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:20.216295 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 07:44:21.216289 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:21.216250 2568 patch_prober.go:28] interesting pod/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 07:44:21.216762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:21.216307 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 07:44:22.219818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:22.219775 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:22.220271 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:22.220069 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:22.220951 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:22.220931 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:27.697384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.697347 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2"] Jul 29 07:44:27.697882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.697794 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" Jul 29 07:44:27.697882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.697807 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" Jul 29 07:44:27.697959 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.697894 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" Jul 29 07:44:27.697959 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.697904 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" Jul 29 07:44:27.701142 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.701124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:27.704543 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.704507 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:44:27.711916 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.711883 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2"] Jul 29 07:44:27.760985 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.760938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3564323f-4v9w2\" (UID: \"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:27.862122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.862080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3564323f-4v9w2\" (UID: \"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:27.862698 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:27.862676 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3564323f-4v9w2\" (UID: \"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:28.013621 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:28.013521 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:28.173357 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:28.173320 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2"] Jul 29 07:44:28.176097 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:44:28.176066 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9445a9f7_bfc1_4374_9e7d_40abc6ed0b6f.slice/crio-b03f605363fb911f88b7b36ab6fc167eb1e78d4c2014e7083119d5d44531b836 WatchSource:0}: Error finding container b03f605363fb911f88b7b36ab6fc167eb1e78d4c2014e7083119d5d44531b836: Status 404 returned error can't find the container with id b03f605363fb911f88b7b36ab6fc167eb1e78d4c2014e7083119d5d44531b836 Jul 29 07:44:28.513995 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:28.513952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" event={"ID":"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f","Type":"ContainerStarted","Data":"c4e9f2adbfd08068221a638e721a6a61279b5c33c9d44b669014232ef8d1a0ec"} Jul 29 07:44:28.514223 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:28.514027 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" event={"ID":"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f","Type":"ContainerStarted","Data":"b03f605363fb911f88b7b36ab6fc167eb1e78d4c2014e7083119d5d44531b836"} Jul 29 07:44:28.535312 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:28.535255 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" podStartSLOduration=1.535234584 podStartE2EDuration="1.535234584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:44:28.534451492 +0000 UTC m=+1778.305902291" watchObservedRunningTime="2026-07-29 07:44:28.535234584 +0000 UTC m=+1778.306685384" Jul 29 07:44:29.521259 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:29.521220 2568 generic.go:356] "Generic (PLEG): container finished" podID="9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f" containerID="c4e9f2adbfd08068221a638e721a6a61279b5c33c9d44b669014232ef8d1a0ec" exitCode=0 Jul 29 07:44:29.521654 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:29.521311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" event={"ID":"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f","Type":"ContainerDied","Data":"c4e9f2adbfd08068221a638e721a6a61279b5c33c9d44b669014232ef8d1a0ec"} Jul 29 07:44:30.672679 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:30.672645 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:30.791777 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:30.791685 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f-maas-api-service-ca\") pod \"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f\" (UID: \"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f\") " Jul 29 07:44:30.792131 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:30.792108 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f" (UID: "9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:44:30.892961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:30.892923 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:31.533860 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:31.533828 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" Jul 29 07:44:31.534079 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:31.533871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2" event={"ID":"9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f","Type":"ContainerDied","Data":"b03f605363fb911f88b7b36ab6fc167eb1e78d4c2014e7083119d5d44531b836"} Jul 29 07:44:31.534079 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:31.533924 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03f605363fb911f88b7b36ab6fc167eb1e78d4c2014e7083119d5d44531b836" Jul 29 07:44:33.564237 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.564186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:33.647688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.647649 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:33.725127 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.725033 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:33.763129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.763093 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2"] Jul 29 07:44:33.766114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.766079 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3564323f-4v9w2"] Jul 29 07:44:33.792583 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.792535 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:33.912496 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.912457 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn"] Jul 29 07:44:33.912748 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.912723 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" containerID="cri-o://53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825" gracePeriod=30 Jul 29 07:44:33.940679 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:33.940638 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:34.253876 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:34.253787 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:34.732959 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:34.732910 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:34.738791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:34.738756 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f" path="/var/lib/kubelet/pods/9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f/volumes" Jul 29 07:44:34.831616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:34.831558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:34.924020 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:34.923982 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:36.443517 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:36.443462 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:37.223394 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:37.223356 2568 patch_prober.go:28] interesting pod/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:44:37.223575 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:37.223416 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:44:39.187619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.187544 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:39.279988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.279948 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-socket\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.279988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.279987 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-istio-token\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280007 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-certs\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280066 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-credential-socket\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280087 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85812904-c299-4c41-991e-a2158b10abec-istiod-ca-cert\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280103 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97pc5\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-kube-api-access-97pc5\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280139 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-envoy\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280283 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280193 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-data\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280621 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280361 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:39.280621 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280400 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:39.280621 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280415 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85812904-c299-4c41-991e-a2158b10abec-istio-podinfo\") pod \"85812904-c299-4c41-991e-a2158b10abec\" (UID: \"85812904-c299-4c41-991e-a2158b10abec\") " Jul 29 07:44:39.280621 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280570 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:39.280813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280627 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85812904-c299-4c41-991e-a2158b10abec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:44:39.280813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280753 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.280813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280772 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85812904-c299-4c41-991e-a2158b10abec-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.280813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280785 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.280813 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.280799 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.281524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.281492 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-data" (OuterVolumeSpecName: "istio-data") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:39.282573 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.282550 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-istio-token" (OuterVolumeSpecName: "istio-token") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:39.282710 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.282689 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85812904-c299-4c41-991e-a2158b10abec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:44:39.283210 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.283188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:44:39.283210 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.283196 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-kube-api-access-97pc5" (OuterVolumeSpecName: "kube-api-access-97pc5") pod "85812904-c299-4c41-991e-a2158b10abec" (UID: "85812904-c299-4c41-991e-a2158b10abec"). InnerVolumeSpecName "kube-api-access-97pc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:39.381680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.381647 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85812904-c299-4c41-991e-a2158b10abec-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.381680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.381677 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.381680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.381688 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97pc5\" (UniqueName: \"kubernetes.io/projected/85812904-c299-4c41-991e-a2158b10abec-kube-api-access-97pc5\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.381928 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.381697 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.381928 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.381705 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85812904-c299-4c41-991e-a2158b10abec-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:44:39.581858 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.581765 2568 generic.go:356] "Generic (PLEG): container finished" podID="85812904-c299-4c41-991e-a2158b10abec" containerID="53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825" exitCode=0 Jul 29 07:44:39.581858 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.581831 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" Jul 29 07:44:39.582140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.581858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" event={"ID":"85812904-c299-4c41-991e-a2158b10abec","Type":"ContainerDied","Data":"53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825"} Jul 29 07:44:39.582140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.581940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn" event={"ID":"85812904-c299-4c41-991e-a2158b10abec","Type":"ContainerDied","Data":"fb5b15fcbbc2db9fe60002d9d2e59c2f77454fa44ea0b01a1e9204b927597485"} Jul 29 07:44:39.582140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.581972 2568 scope.go:117] "RemoveContainer" containerID="53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825" Jul 29 07:44:39.592876 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.592854 2568 scope.go:117] "RemoveContainer" containerID="f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962" Jul 29 07:44:39.602461 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.602440 2568 scope.go:117] "RemoveContainer" containerID="53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825" Jul 29 07:44:39.602816 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:44:39.602793 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825\": container with ID starting with 53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825 not found: ID does not exist" containerID="53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825" Jul 29 07:44:39.602955 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.602825 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825"} err="failed to get container status \"53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825\": rpc error: code = NotFound desc = could not find container \"53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825\": container with ID starting with 53b4dd3c40b41e98d5c8662ccb30131848adb4ac062ac754567b4f51a514c825 not found: ID does not exist" Jul 29 07:44:39.602955 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.602845 2568 scope.go:117] "RemoveContainer" containerID="f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962" Jul 29 07:44:39.603132 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:44:39.603109 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962\": container with ID starting with f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962 not found: ID does not exist" containerID="f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962" Jul 29 07:44:39.603196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.603152 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962"} err="failed to get container status \"f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962\": rpc error: code = NotFound desc = could not find container \"f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962\": container with ID starting with f3be01a2739ac75ccbdbe79504330cd1962c619d098bffdeba89bfacd6604962 not found: ID does not exist" Jul 29 07:44:39.615147 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.615110 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn"] Jul 29 07:44:39.622059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:39.622013 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3564323f-openshift-default-64b5755ddd-pvdkn"] Jul 29 07:44:40.738530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:40.738492 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85812904-c299-4c41-991e-a2158b10abec" path="/var/lib/kubelet/pods/85812904-c299-4c41-991e-a2158b10abec/volumes" Jul 29 07:44:43.280179 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:43.280083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:43.399019 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:43.398974 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:43.544834 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:43.544757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:43.763936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:43.763897 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:44.163275 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:44.163238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:44:50.770843 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:50.770813 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:44:50.783973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:44:50.783945 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:45:00.136926 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.136890 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq"] Jul 29 07:45:00.137415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137328 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" Jul 29 07:45:00.137415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137342 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" Jul 29 07:45:00.137415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137364 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" Jul 29 07:45:00.137415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137369 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d443172f-397e-4e3c-9558-8e2b54b78726" containerName="istio-proxy" Jul 29 07:45:00.137415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137376 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f" containerName="revoke-keys" Jul 29 07:45:00.137415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137381 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f" containerName="revoke-keys" Jul 29 07:45:00.137608 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137447 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9445a9f7-bfc1-4374-9e7d-40abc6ed0b6f" containerName="revoke-keys" Jul 29 07:45:00.137608 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.137456 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" Jul 29 07:45:00.141786 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.141768 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" Jul 29 07:45:00.144767 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.144739 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:45:00.146882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.146858 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq"] Jul 29 07:45:00.154191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.154170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" Jul 29 07:45:00.505855 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.505827 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq"] Jul 29 07:45:00.508621 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:45:00.508590 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400f0ee4_df5f_4706_ab11_7b405a60d0a8.slice/crio-764003e831a8fb68520c08bc6bcd83e2e69a6f3dfd7d2a193dd1175bc3acd181 WatchSource:0}: Error finding container 764003e831a8fb68520c08bc6bcd83e2e69a6f3dfd7d2a193dd1175bc3acd181: Status 404 returned error can't find the container with id 764003e831a8fb68520c08bc6bcd83e2e69a6f3dfd7d2a193dd1175bc3acd181 Jul 29 07:45:00.700599 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:00.700563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" event={"ID":"400f0ee4-df5f-4706-ab11-7b405a60d0a8","Type":"ContainerStarted","Data":"764003e831a8fb68520c08bc6bcd83e2e69a6f3dfd7d2a193dd1175bc3acd181"} Jul 29 07:45:01.707196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:01.707106 2568 generic.go:356] "Generic (PLEG): container finished" podID="400f0ee4-df5f-4706-ab11-7b405a60d0a8" containerID="87edcf4667ae4f9d88bf73cbaf2b370dafe65c0244597a30f7adc2791a65a902" exitCode=0 Jul 29 07:45:01.707549 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:01.707194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" event={"ID":"400f0ee4-df5f-4706-ab11-7b405a60d0a8","Type":"ContainerDied","Data":"87edcf4667ae4f9d88bf73cbaf2b370dafe65c0244597a30f7adc2791a65a902"} Jul 29 07:45:02.855195 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:02.855160 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" Jul 29 07:45:03.719294 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:03.719263 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" Jul 29 07:45:03.719505 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:03.719263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755185-dj2cq" event={"ID":"400f0ee4-df5f-4706-ab11-7b405a60d0a8","Type":"ContainerDied","Data":"764003e831a8fb68520c08bc6bcd83e2e69a6f3dfd7d2a193dd1175bc3acd181"} Jul 29 07:45:03.719505 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:03.719427 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764003e831a8fb68520c08bc6bcd83e2e69a6f3dfd7d2a193dd1175bc3acd181" Jul 29 07:45:10.082256 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:10.082216 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:17.435161 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:17.435106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:17.533854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:17.533822 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:17.658886 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:17.658849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:33.646401 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:33.646361 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:39.389697 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:39.389658 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:39.499888 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:39.499854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:39.591705 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:39.591670 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:50.143715 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:50.143674 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:51.042975 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:51.042926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:58.312249 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:58.312205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:58.423902 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:58.423866 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:58.510613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:58.510568 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:45:58.612384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:45:58.612278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:03.759640 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:03.759596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:03.855722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:03.855685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:03.955806 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:03.955771 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:05.766213 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:05.766175 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:11.918925 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:11.918832 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:12.004417 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:12.004382 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:12.688223 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:12.688181 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:18.577384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:18.577348 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:18.682005 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:18.681947 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:18.826659 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:18.826628 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:18.927319 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:18.927280 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:19.114854 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.114814 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm"] Jul 29 07:46:19.115500 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.115482 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400f0ee4-df5f-4706-ab11-7b405a60d0a8" containerName="cleanup" Jul 29 07:46:19.115566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.115505 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="400f0ee4-df5f-4706-ab11-7b405a60d0a8" containerName="cleanup" Jul 29 07:46:19.115566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.115552 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" Jul 29 07:46:19.115566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.115561 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" Jul 29 07:46:19.115670 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.115662 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85812904-c299-4c41-991e-a2158b10abec" containerName="istio-proxy" Jul 29 07:46:19.115722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.115681 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="400f0ee4-df5f-4706-ab11-7b405a60d0a8" containerName="cleanup" Jul 29 07:46:19.119338 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.119308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.122971 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.122610 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e2c7c574-openshift-default-dockercfg-k27p6\"" Jul 29 07:46:19.122971 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.122901 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:46:19.137484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.136867 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm"] Jul 29 07:46:19.151640 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.151605 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:19.299426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299323 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fkv\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-kube-api-access-t2fkv\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-podinfo\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-data\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-certs\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-token\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-socket\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299634 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-envoy\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299650 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istiod-ca-cert\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.299860 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.299716 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-credential-socket\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.339029 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.338986 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:19.400349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-envoy\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istiod-ca-cert\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400627 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-credential-socket\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400627 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fkv\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-kube-api-access-t2fkv\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400627 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-podinfo\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400627 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-data\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-certs\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-token\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.400846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-socket\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.401009 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-credential-socket\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.401009 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.400914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-data\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.401261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.401230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-socket\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.401362 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.401338 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-certs\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.401362 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.401349 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istiod-ca-cert\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.402897 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.402873 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-envoy\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.403254 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.403235 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-podinfo\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.413240 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.413205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fkv\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-kube-api-access-t2fkv\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.416087 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.416022 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-token\") pod \"e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.446929 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.446877 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:19.616891 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.616769 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm"] Jul 29 07:46:19.619611 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:46:19.619574 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11c44c2_ddbd_44f3_83ea_ae4538a51da9.slice/crio-73230af1a68d98dfdf26ed3ac85ca43d6b958d739c07a7adceed2243abb06e71 WatchSource:0}: Error finding container 73230af1a68d98dfdf26ed3ac85ca43d6b958d739c07a7adceed2243abb06e71: Status 404 returned error can't find the container with id 73230af1a68d98dfdf26ed3ac85ca43d6b958d739c07a7adceed2243abb06e71 Jul 29 07:46:19.622333 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.622284 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:46:19.622484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.622390 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:46:19.622484 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:19.622435 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:46:20.155231 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:20.155196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" event={"ID":"e11c44c2-ddbd-44f3-83ea-ae4538a51da9","Type":"ContainerStarted","Data":"25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76"} Jul 29 07:46:20.155467 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:20.155260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" event={"ID":"e11c44c2-ddbd-44f3-83ea-ae4538a51da9","Type":"ContainerStarted","Data":"73230af1a68d98dfdf26ed3ac85ca43d6b958d739c07a7adceed2243abb06e71"} Jul 29 07:46:20.183574 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:20.183503 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" podStartSLOduration=1.183482906 podStartE2EDuration="1.183482906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:46:20.1820787 +0000 UTC m=+1889.953529496" watchObservedRunningTime="2026-07-29 07:46:20.183482906 +0000 UTC m=+1889.954933704" Jul 29 07:46:20.447794 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:20.447696 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:20.453212 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:20.453180 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:21.162932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:21.162890 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:21.163934 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:21.163904 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:22.266315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:22.266263 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:22.386950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:22.386914 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:22.487753 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:22.487709 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:23.053093 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:23.053027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:23.154651 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:23.154621 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:33.234749 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:33.234711 2568 generic.go:356] "Generic (PLEG): container finished" podID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerID="25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76" exitCode=0 Jul 29 07:46:33.235337 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:33.234791 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" event={"ID":"e11c44c2-ddbd-44f3-83ea-ae4538a51da9","Type":"ContainerDied","Data":"25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76"} Jul 29 07:46:33.235410 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:33.235332 2568 scope.go:117] "RemoveContainer" containerID="25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76" Jul 29 07:46:33.236612 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:33.236581 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:46:33.236752 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:33.236691 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:46:33.236752 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:33.236727 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:46:34.246903 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:34.246851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" event={"ID":"e11c44c2-ddbd-44f3-83ea-ae4538a51da9","Type":"ContainerStarted","Data":"f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e"} Jul 29 07:46:34.447520 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:34.447475 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:34.448942 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:34.448917 2568 patch_prober.go:28] interesting pod/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 07:46:34.449089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:34.448978 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 07:46:35.447969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:35.447917 2568 patch_prober.go:28] interesting pod/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 07:46:35.448424 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:35.448001 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 07:46:36.163059 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:36.163007 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:36.447653 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:36.447546 2568 patch_prober.go:28] interesting pod/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 07:46:36.447970 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:36.447938 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 07:46:37.447849 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:37.447800 2568 patch_prober.go:28] interesting pod/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 07:46:37.448320 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:37.447876 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 07:46:38.452577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:38.452544 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:38.453812 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:38.453792 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:46:47.294386 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:47.294346 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:48.006683 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:48.006652 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:48.106846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:48.106812 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:48.482600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:48.482565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:54.981423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:54.981385 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:55.081950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.081911 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:46:55.361830 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.361744 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn"] Jul 29 07:46:55.364306 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.364286 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:55.367495 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.367469 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:46:55.380540 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.380499 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn"] Jul 29 07:46:55.440571 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.440529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7806a33-e18d-4e40-82f2-700a2efadc37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn\" (UID: \"f7806a33-e18d-4e40-82f2-700a2efadc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:55.541558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.541514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7806a33-e18d-4e40-82f2-700a2efadc37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn\" (UID: \"f7806a33-e18d-4e40-82f2-700a2efadc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:55.542247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.542225 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7806a33-e18d-4e40-82f2-700a2efadc37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn\" (UID: \"f7806a33-e18d-4e40-82f2-700a2efadc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:55.684820 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.684781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:55.841216 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:55.841167 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn"] Jul 29 07:46:55.843353 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:46:55.843326 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7806a33_e18d_4e40_82f2_700a2efadc37.slice/crio-3a091e512386e66fe7f6d6d4ff518512cd8114d9ad28dee01c04661121d30bb2 WatchSource:0}: Error finding container 3a091e512386e66fe7f6d6d4ff518512cd8114d9ad28dee01c04661121d30bb2: Status 404 returned error can't find the container with id 3a091e512386e66fe7f6d6d4ff518512cd8114d9ad28dee01c04661121d30bb2 Jul 29 07:46:56.367173 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:56.367083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" event={"ID":"f7806a33-e18d-4e40-82f2-700a2efadc37","Type":"ContainerStarted","Data":"ba6c2f6beeeb99353fd0d938e1995157e49909ac277d7baf6e6aae41204a2654"} Jul 29 07:46:56.367173 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:56.367138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" event={"ID":"f7806a33-e18d-4e40-82f2-700a2efadc37","Type":"ContainerStarted","Data":"3a091e512386e66fe7f6d6d4ff518512cd8114d9ad28dee01c04661121d30bb2"} Jul 29 07:46:56.383295 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:56.383217 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" podStartSLOduration=1.3831967299999999 podStartE2EDuration="1.38319673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:46:56.382230594 +0000 UTC m=+1926.153681395" watchObservedRunningTime="2026-07-29 07:46:56.38319673 +0000 UTC m=+1926.154647527" Jul 29 07:46:57.374440 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:57.374388 2568 generic.go:356] "Generic (PLEG): container finished" podID="f7806a33-e18d-4e40-82f2-700a2efadc37" containerID="ba6c2f6beeeb99353fd0d938e1995157e49909ac277d7baf6e6aae41204a2654" exitCode=0 Jul 29 07:46:57.374916 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:57.374445 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" event={"ID":"f7806a33-e18d-4e40-82f2-700a2efadc37","Type":"ContainerDied","Data":"ba6c2f6beeeb99353fd0d938e1995157e49909ac277d7baf6e6aae41204a2654"} Jul 29 07:46:58.522006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:58.521977 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:58.671893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:58.671852 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7806a33-e18d-4e40-82f2-700a2efadc37-maas-api-service-ca\") pod \"f7806a33-e18d-4e40-82f2-700a2efadc37\" (UID: \"f7806a33-e18d-4e40-82f2-700a2efadc37\") " Jul 29 07:46:58.672247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:58.672223 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7806a33-e18d-4e40-82f2-700a2efadc37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7806a33-e18d-4e40-82f2-700a2efadc37" (UID: "f7806a33-e18d-4e40-82f2-700a2efadc37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:46:58.773150 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:58.773113 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7806a33-e18d-4e40-82f2-700a2efadc37-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:46:59.386489 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:59.386453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" event={"ID":"f7806a33-e18d-4e40-82f2-700a2efadc37","Type":"ContainerDied","Data":"3a091e512386e66fe7f6d6d4ff518512cd8114d9ad28dee01c04661121d30bb2"} Jul 29 07:46:59.386489 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:59.386489 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn" Jul 29 07:46:59.386709 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:46:59.386505 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a091e512386e66fe7f6d6d4ff518512cd8114d9ad28dee01c04661121d30bb2" Jul 29 07:47:05.952418 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:05.952362 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:06.035939 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.035903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:06.104144 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.104106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:06.164225 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.164183 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn"] Jul 29 07:47:06.167184 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.167158 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2c7c574-p92cn"] Jul 29 07:47:06.186286 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.186245 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:06.622004 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.621960 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:06.739414 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:06.739372 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7806a33-e18d-4e40-82f2-700a2efadc37" path="/var/lib/kubelet/pods/f7806a33-e18d-4e40-82f2-700a2efadc37/volumes" Jul 29 07:47:13.739239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:13.739205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:13.901167 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:13.901133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:14.148267 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.148178 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm"] Jul 29 07:47:14.148421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.148399 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" containerID="cri-o://f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e" gracePeriod=30 Jul 29 07:47:14.198449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.198410 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:14.421369 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.421308 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:14.592349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.592306 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:14.715130 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.714999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:14.815904 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:14.815866 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:19.410338 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.410267 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:47:19.463342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463308 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-token\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463349 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-data\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463370 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-certs\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2fkv\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-kube-api-access-t2fkv\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463425 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-podinfo\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463451 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-socket\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463493 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-envoy\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463516 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-credential-socket\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463600 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463560 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istiod-ca-cert\") pod \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\" (UID: \"e11c44c2-ddbd-44f3-83ea-ae4538a51da9\") " Jul 29 07:47:19.463962 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463711 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:47:19.463962 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463892 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.463962 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.463906 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:47:19.464239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.464090 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:47:19.464239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.464180 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:47:19.464457 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.464278 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-data" (OuterVolumeSpecName: "istio-data") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:47:19.465862 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.465836 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-token" (OuterVolumeSpecName: "istio-token") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:47:19.466017 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.465990 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:47:19.466341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.466317 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:47:19.466421 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.466400 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-kube-api-access-t2fkv" (OuterVolumeSpecName: "kube-api-access-t2fkv") pod "e11c44c2-ddbd-44f3-83ea-ae4538a51da9" (UID: "e11c44c2-ddbd-44f3-83ea-ae4538a51da9"). InnerVolumeSpecName "kube-api-access-t2fkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:47:19.499150 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.499114 2568 generic.go:356] "Generic (PLEG): container finished" podID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerID="f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e" exitCode=0 Jul 29 07:47:19.499342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.499170 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" Jul 29 07:47:19.499342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.499194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" event={"ID":"e11c44c2-ddbd-44f3-83ea-ae4538a51da9","Type":"ContainerDied","Data":"f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e"} Jul 29 07:47:19.499342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.499250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm" event={"ID":"e11c44c2-ddbd-44f3-83ea-ae4538a51da9","Type":"ContainerDied","Data":"73230af1a68d98dfdf26ed3ac85ca43d6b958d739c07a7adceed2243abb06e71"} Jul 29 07:47:19.499342 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.499275 2568 scope.go:117] "RemoveContainer" containerID="f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e" Jul 29 07:47:19.510055 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.510018 2568 scope.go:117] "RemoveContainer" containerID="25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76" Jul 29 07:47:19.519814 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.519785 2568 scope.go:117] "RemoveContainer" containerID="f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e" Jul 29 07:47:19.520303 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:47:19.520277 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e\": container with ID starting with f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e not found: ID does not exist" containerID="f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e" Jul 29 07:47:19.520429 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.520310 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e"} err="failed to get container status \"f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e\": rpc error: code = NotFound desc = could not find container \"f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e\": container with ID starting with f14f9a88a90629862ffbee859f5c9825767d4a4ba1d65fdbca445c481c73d74e not found: ID does not exist" Jul 29 07:47:19.520429 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.520331 2568 scope.go:117] "RemoveContainer" containerID="25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76" Jul 29 07:47:19.520930 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:47:19.520897 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76\": container with ID starting with 25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76 not found: ID does not exist" containerID="25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76" Jul 29 07:47:19.521087 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.520943 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76"} err="failed to get container status \"25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76\": rpc error: code = NotFound desc = could not find container \"25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76\": container with ID starting with 25451f7a715a2e65303ebfb357b7cf71645e2ad9584bd93b82ea5a462b56fd76 not found: ID does not exist" Jul 29 07:47:19.523368 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.523340 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm"] Jul 29 07:47:19.527489 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.527455 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2c7c574-openshift-default-c85fd9bb4-zbfwm"] Jul 29 07:47:19.564677 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564634 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564677 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564668 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564677 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564677 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fkv\" (UniqueName: \"kubernetes.io/projected/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-kube-api-access-t2fkv\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564677 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564688 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564967 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564702 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564967 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564711 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564967 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564720 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:19.564967 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:19.564727 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11c44c2-ddbd-44f3-83ea-ae4538a51da9-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:47:20.744809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:20.744764 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" path="/var/lib/kubelet/pods/e11c44c2-ddbd-44f3-83ea-ae4538a51da9/volumes" Jul 29 07:47:20.814360 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:20.814318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:20.978529 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:20.978492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:21.116978 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:21.116897 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:21.314875 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:21.314840 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:24.477605 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:24.477571 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:24.594413 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:24.594377 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:28.784133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:28.784101 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:28.977716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:28.977686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:31.729492 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:31.729457 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:38.518657 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:38.518616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:39.152575 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152538 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b"] Jul 29 07:47:39.152938 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152926 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" Jul 29 07:47:39.153001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152940 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" Jul 29 07:47:39.153001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152949 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" Jul 29 07:47:39.153001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152955 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" Jul 29 07:47:39.153001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152962 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7806a33-e18d-4e40-82f2-700a2efadc37" containerName="revoke-keys" Jul 29 07:47:39.153001 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.152968 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7806a33-e18d-4e40-82f2-700a2efadc37" containerName="revoke-keys" Jul 29 07:47:39.153178 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.153070 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" Jul 29 07:47:39.153178 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.153080 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7806a33-e18d-4e40-82f2-700a2efadc37" containerName="revoke-keys" Jul 29 07:47:39.155362 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.155343 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.159979 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.159954 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-412a3f1a-openshift-default-dockercfg-rct8l\"" Jul 29 07:47:39.162264 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.162240 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:47:39.202463 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.202427 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b"] Jul 29 07:47:39.207885 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.207854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:39.246234 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-credential-socket\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246414 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-socket\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246414 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246301 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-envoy\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246414 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246354 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-certs\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246414 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgl7\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-kube-api-access-vbgl7\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246563 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246455 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-istio-token\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246563 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246480 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf943cf-f690-4a50-902d-be206e9d903d-istiod-ca-cert\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246563 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-data\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.246563 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.246525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf943cf-f690-4a50-902d-be206e9d903d-istio-podinfo\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.347774 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-data\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.347774 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf943cf-f690-4a50-902d-be206e9d903d-istio-podinfo\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-credential-socket\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-socket\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-envoy\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-certs\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgl7\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-kube-api-access-vbgl7\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347956 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-istio-token\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.347987 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf943cf-f690-4a50-902d-be206e9d903d-istiod-ca-cert\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348444 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.348229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-data\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348444 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.348283 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-credential-socket\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348444 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.348423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-certs\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348610 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.348494 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-socket\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.348875 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.348850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf943cf-f690-4a50-902d-be206e9d903d-istiod-ca-cert\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.350510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.350491 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-envoy\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.350576 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.350559 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf943cf-f690-4a50-902d-be206e9d903d-istio-podinfo\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.370560 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.370525 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-istio-token\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.370983 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.370962 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgl7\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-kube-api-access-vbgl7\") pod \"e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.467012 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.466911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:39.586472 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.586427 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:39.651649 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.651611 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b"] Jul 29 07:47:39.652414 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:47:39.652388 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf943cf_f690_4a50_902d_be206e9d903d.slice/crio-ee74bae2defa6d0f1b560ede040ae2e5f5d15bda304dacecd351fd2286040ced WatchSource:0}: Error finding container ee74bae2defa6d0f1b560ede040ae2e5f5d15bda304dacecd351fd2286040ced: Status 404 returned error can't find the container with id ee74bae2defa6d0f1b560ede040ae2e5f5d15bda304dacecd351fd2286040ced Jul 29 07:47:39.654793 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.654744 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:47:39.654893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.654826 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:47:39.654893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:39.654856 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:47:40.618960 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:40.618913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" event={"ID":"baf943cf-f690-4a50-902d-be206e9d903d","Type":"ContainerStarted","Data":"6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d"} Jul 29 07:47:40.619397 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:40.618978 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" event={"ID":"baf943cf-f690-4a50-902d-be206e9d903d","Type":"ContainerStarted","Data":"ee74bae2defa6d0f1b560ede040ae2e5f5d15bda304dacecd351fd2286040ced"} Jul 29 07:47:41.467865 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:41.467767 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:41.473336 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:41.473305 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:41.551461 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:41.551394 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podStartSLOduration=2.551377447 podStartE2EDuration="2.551377447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:47:40.689281381 +0000 UTC m=+1970.460732182" watchObservedRunningTime="2026-07-29 07:47:41.551377447 +0000 UTC m=+1971.322828246" Jul 29 07:47:41.625294 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:41.625246 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:41.626330 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:41.626305 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:47:43.171623 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:43.171590 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:43.452614 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:43.452530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:43.725178 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:43.725095 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:43.891778 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:43.891745 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:45.102301 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:45.102266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:45.318238 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:45.318208 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:52.722898 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:52.722818 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:47:57.666249 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:57.666204 2568 patch_prober.go:28] interesting pod/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:47:57.666666 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:57.666279 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:47:58.717144 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:58.717111 2568 generic.go:356] "Generic (PLEG): container finished" podID="baf943cf-f690-4a50-902d-be206e9d903d" containerID="6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d" exitCode=0 Jul 29 07:47:58.717649 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:58.717182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" event={"ID":"baf943cf-f690-4a50-902d-be206e9d903d","Type":"ContainerDied","Data":"6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d"} Jul 29 07:47:58.717649 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:58.717610 2568 scope.go:117] "RemoveContainer" containerID="6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d" Jul 29 07:47:58.718669 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:58.718642 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:47:58.718732 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:58.718710 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:47:58.718772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:58.718739 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:47:59.727927 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:47:59.727878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" event={"ID":"baf943cf-f690-4a50-902d-be206e9d903d","Type":"ContainerStarted","Data":"39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0"} Jul 29 07:48:00.467833 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:00.467789 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:48:00.468173 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:00.468139 2568 patch_prober.go:28] interesting pod/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 07:48:00.468242 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:00.468194 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 07:48:01.468182 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:01.468138 2568 patch_prober.go:28] interesting pod/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 07:48:01.468616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:01.468199 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 07:48:02.467566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:02.467521 2568 patch_prober.go:28] interesting pod/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 07:48:02.467772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:02.467591 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 07:48:03.468133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:03.468093 2568 patch_prober.go:28] interesting pod/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 07:48:03.468566 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:03.468151 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 07:48:04.241831 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:04.241790 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:48:04.471452 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:04.471418 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:48:04.471938 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:04.471755 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:48:04.472534 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:04.472511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:48:33.923735 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:33.923695 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b"] Jul 29 07:48:33.924281 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:33.924226 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11c44c2-ddbd-44f3-83ea-ae4538a51da9" containerName="istio-proxy" Jul 29 07:48:33.926411 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:33.926393 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:33.930115 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:33.930097 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:48:33.959821 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:33.959784 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b"] Jul 29 07:48:34.062136 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.062096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f73e9ef-2a17-41b2-baea-2183ed4a89a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b\" (UID: \"1f73e9ef-2a17-41b2-baea-2183ed4a89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:34.163501 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.163461 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f73e9ef-2a17-41b2-baea-2183ed4a89a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b\" (UID: \"1f73e9ef-2a17-41b2-baea-2183ed4a89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:34.164183 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.164160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f73e9ef-2a17-41b2-baea-2183ed4a89a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b\" (UID: \"1f73e9ef-2a17-41b2-baea-2183ed4a89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:34.237581 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.237477 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:34.419286 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.419241 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b"] Jul 29 07:48:34.422855 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:48:34.422824 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f73e9ef_2a17_41b2_baea_2183ed4a89a2.slice/crio-5b9d5aab99613ec6d223cfcab811ff8e1a518edc8ea8785a5da5e6509f65304d WatchSource:0}: Error finding container 5b9d5aab99613ec6d223cfcab811ff8e1a518edc8ea8785a5da5e6509f65304d: Status 404 returned error can't find the container with id 5b9d5aab99613ec6d223cfcab811ff8e1a518edc8ea8785a5da5e6509f65304d Jul 29 07:48:34.968397 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.968357 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" event={"ID":"1f73e9ef-2a17-41b2-baea-2183ed4a89a2","Type":"ContainerStarted","Data":"f1a7ce84aa60104f9effb88cc50d9c916f820c7e699f1594ab3c76950ecb7b43"} Jul 29 07:48:34.968815 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:34.968421 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" event={"ID":"1f73e9ef-2a17-41b2-baea-2183ed4a89a2","Type":"ContainerStarted","Data":"5b9d5aab99613ec6d223cfcab811ff8e1a518edc8ea8785a5da5e6509f65304d"} Jul 29 07:48:35.004495 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:35.004441 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" podStartSLOduration=2.004423305 podStartE2EDuration="2.004423305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:48:35.002393572 +0000 UTC m=+2024.773844375" watchObservedRunningTime="2026-07-29 07:48:35.004423305 +0000 UTC m=+2024.775874104" Jul 29 07:48:35.975390 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:35.975352 2568 generic.go:356] "Generic (PLEG): container finished" podID="1f73e9ef-2a17-41b2-baea-2183ed4a89a2" containerID="f1a7ce84aa60104f9effb88cc50d9c916f820c7e699f1594ab3c76950ecb7b43" exitCode=0 Jul 29 07:48:35.975795 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:35.975444 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" event={"ID":"1f73e9ef-2a17-41b2-baea-2183ed4a89a2","Type":"ContainerDied","Data":"f1a7ce84aa60104f9effb88cc50d9c916f820c7e699f1594ab3c76950ecb7b43"} Jul 29 07:48:37.126613 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.126583 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:37.292960 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.292855 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f73e9ef-2a17-41b2-baea-2183ed4a89a2-maas-api-service-ca\") pod \"1f73e9ef-2a17-41b2-baea-2183ed4a89a2\" (UID: \"1f73e9ef-2a17-41b2-baea-2183ed4a89a2\") " Jul 29 07:48:37.293299 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.293270 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f73e9ef-2a17-41b2-baea-2183ed4a89a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f73e9ef-2a17-41b2-baea-2183ed4a89a2" (UID: "1f73e9ef-2a17-41b2-baea-2183ed4a89a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:48:37.393680 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.393642 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f73e9ef-2a17-41b2-baea-2183ed4a89a2-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:48:37.997032 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.996987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" event={"ID":"1f73e9ef-2a17-41b2-baea-2183ed4a89a2","Type":"ContainerDied","Data":"5b9d5aab99613ec6d223cfcab811ff8e1a518edc8ea8785a5da5e6509f65304d"} Jul 29 07:48:37.997247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.997030 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b" Jul 29 07:48:37.997247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:37.997100 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b9d5aab99613ec6d223cfcab811ff8e1a518edc8ea8785a5da5e6509f65304d" Jul 29 07:48:45.537695 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:45.537660 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:48:46.054507 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:46.054469 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:48:46.183426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:46.183381 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b"] Jul 29 07:48:46.220900 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:46.220859 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-412a3f1a-lzb4b"] Jul 29 07:48:46.402424 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:46.402369 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:48:46.739527 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:46.739435 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f73e9ef-2a17-41b2-baea-2183ed4a89a2" path="/var/lib/kubelet/pods/1f73e9ef-2a17-41b2-baea-2183ed4a89a2/volumes" Jul 29 07:48:50.514503 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:50.514432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:48:50.962574 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:50.962537 2568 scope.go:117] "RemoveContainer" containerID="f10ac0e8c10afb791fc69d5beb684183f37491f6096470dabfa7f75b82758ca1" Jul 29 07:48:55.668436 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:55.668391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:48:58.105091 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:48:58.105017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:32.512669 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:32.512625 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:34.385882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:34.385827 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:41.135338 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:41.135304 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:41.237247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:41.237187 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:41.446762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:41.446714 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b"] Jul 29 07:49:41.447019 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:41.446961 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" containerID="cri-o://39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0" gracePeriod=30 Jul 29 07:49:41.467374 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:41.467317 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:41.779259 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:41.779172 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:42.959259 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:42.959204 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:43.055685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:43.055654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:44.118417 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:44.118383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:44.217427 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:44.217384 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:45.846189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:45.846146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:46.721233 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.721157 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:49:46.910929 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.910894 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-credential-socket\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.910929 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.910932 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-certs\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.910972 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-envoy\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911079 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf943cf-f690-4a50-902d-be206e9d903d-istiod-ca-cert\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911114 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-data\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911149 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-istio-token\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911196 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-socket\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911249 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbgl7\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-kube-api-access-vbgl7\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911276 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf943cf-f690-4a50-902d-be206e9d903d-istio-podinfo\") pod \"baf943cf-f690-4a50-902d-be206e9d903d\" (UID: \"baf943cf-f690-4a50-902d-be206e9d903d\") " Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911310 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911323 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:49:46.911524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911492 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf943cf-f690-4a50-902d-be206e9d903d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:49:46.912232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911553 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf943cf-f690-4a50-902d-be206e9d903d-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:46.912232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911573 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:46.912232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911588 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:46.912232 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.911663 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:49:46.912716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.912688 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-data" (OuterVolumeSpecName: "istio-data") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:49:46.913678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.913636 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:49:46.913875 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.913849 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-istio-token" (OuterVolumeSpecName: "istio-token") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:49:46.913933 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.913866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-kube-api-access-vbgl7" (OuterVolumeSpecName: "kube-api-access-vbgl7") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "kube-api-access-vbgl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:49:46.914262 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:46.914239 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baf943cf-f690-4a50-902d-be206e9d903d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "baf943cf-f690-4a50-902d-be206e9d903d" (UID: "baf943cf-f690-4a50-902d-be206e9d903d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:49:47.012932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.012836 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:47.012932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.012869 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:47.012932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.012880 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:47.012932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.012892 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbgl7\" (UniqueName: \"kubernetes.io/projected/baf943cf-f690-4a50-902d-be206e9d903d-kube-api-access-vbgl7\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:47.012932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.012902 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf943cf-f690-4a50-902d-be206e9d903d-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:47.012932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.012912 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf943cf-f690-4a50-902d-be206e9d903d-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:49:47.390887 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.390794 2568 generic.go:356] "Generic (PLEG): container finished" podID="baf943cf-f690-4a50-902d-be206e9d903d" containerID="39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0" exitCode=0 Jul 29 07:49:47.390887 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.390847 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" Jul 29 07:49:47.391110 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.390886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" event={"ID":"baf943cf-f690-4a50-902d-be206e9d903d","Type":"ContainerDied","Data":"39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0"} Jul 29 07:49:47.391110 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.390948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b" event={"ID":"baf943cf-f690-4a50-902d-be206e9d903d","Type":"ContainerDied","Data":"ee74bae2defa6d0f1b560ede040ae2e5f5d15bda304dacecd351fd2286040ced"} Jul 29 07:49:47.391110 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.390979 2568 scope.go:117] "RemoveContainer" containerID="39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0" Jul 29 07:49:47.401964 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.401943 2568 scope.go:117] "RemoveContainer" containerID="6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d" Jul 29 07:49:47.414714 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.414689 2568 scope.go:117] "RemoveContainer" containerID="39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0" Jul 29 07:49:47.415113 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:49:47.415082 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0\": container with ID starting with 39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0 not found: ID does not exist" containerID="39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0" Jul 29 07:49:47.415245 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.415126 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0"} err="failed to get container status \"39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0\": rpc error: code = NotFound desc = could not find container \"39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0\": container with ID starting with 39a6ffbd98c44a8f5c7b6f1df59c07911213b711b6a8bdc0373128a0779f64c0 not found: ID does not exist" Jul 29 07:49:47.415245 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.415150 2568 scope.go:117] "RemoveContainer" containerID="6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d" Jul 29 07:49:47.415642 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:49:47.415616 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d\": container with ID starting with 6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d not found: ID does not exist" containerID="6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d" Jul 29 07:49:47.415742 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.415646 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d"} err="failed to get container status \"6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d\": rpc error: code = NotFound desc = could not find container \"6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d\": container with ID starting with 6da4caca232ed32d441b1da1b1fbb28317233e8d2c375f5300af643bb37a521d not found: ID does not exist" Jul 29 07:49:47.500191 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.500152 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b"] Jul 29 07:49:47.513829 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:47.513793 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-412a3f1a-openshift-default-778d878f76-s4t5b"] Jul 29 07:49:48.738945 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:48.738907 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf943cf-f690-4a50-902d-be206e9d903d" path="/var/lib/kubelet/pods/baf943cf-f690-4a50-902d-be206e9d903d/volumes" Jul 29 07:49:50.803909 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:50.803884 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:49:50.818292 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:50.818260 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:49:50.985353 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:50.985319 2568 scope.go:117] "RemoveContainer" containerID="3b740beeb087299419ab3f7527b2e54fc4bf89211283ef5cb598759a5e5df4ce" Jul 29 07:49:50.994771 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:50.994750 2568 scope.go:117] "RemoveContainer" containerID="248288df7ba7b1dd4d85615bf9c9bbd3dcb6fad47bb68a2425c135df5b0f9c9f" Jul 29 07:49:52.455728 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:52.455689 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:52.532450 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:52.532418 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:49:53.210161 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:49:53.210130 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:01.058953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:01.058904 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:01.246688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:01.246641 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:01.392099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:01.391984 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:11.898784 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:11.898738 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:12.271318 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:12.271289 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:33.376486 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:33.376448 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:42.091464 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:42.091377 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:43.503395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:43.503357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:43.844018 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:43.843935 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:44.547855 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.547816 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5"] Jul 29 07:50:44.548311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548272 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" Jul 29 07:50:44.548311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548287 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" Jul 29 07:50:44.548311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548310 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" Jul 29 07:50:44.548419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548316 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" Jul 29 07:50:44.548419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548322 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f73e9ef-2a17-41b2-baea-2183ed4a89a2" containerName="revoke-keys" Jul 29 07:50:44.548419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548328 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f73e9ef-2a17-41b2-baea-2183ed4a89a2" containerName="revoke-keys" Jul 29 07:50:44.548419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548397 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" Jul 29 07:50:44.548419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.548410 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f73e9ef-2a17-41b2-baea-2183ed4a89a2" containerName="revoke-keys" Jul 29 07:50:44.551688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.551668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.561404 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.561380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7d6491-openshift-default-dockercfg-zmrf7\"" Jul 29 07:50:44.564937 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.564905 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:50:44.617201 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.617166 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5"] Jul 29 07:50:44.702119 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.702083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:44.710193 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-socket\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-podinfo\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710240 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-credential-socket\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-envoy\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbvv\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-kube-api-access-5qbvv\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-data\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-token\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a672b-e656-4d64-8dac-f04098b8a044-istiod-ca-cert\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.710577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.710432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-certs\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.810982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-data\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-token\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a672b-e656-4d64-8dac-f04098b8a044-istiod-ca-cert\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-certs\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-socket\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-podinfo\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-credential-socket\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-envoy\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbvv\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-kube-api-access-5qbvv\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811457 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-data\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.811726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811669 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-certs\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.812010 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.811975 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a672b-e656-4d64-8dac-f04098b8a044-istiod-ca-cert\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.812010 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.812002 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-socket\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.812219 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.812099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-credential-socket\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.813988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.813962 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-envoy\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.814229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.814211 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-podinfo\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.852587 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.852551 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbvv\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-kube-api-access-5qbvv\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.858333 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.858306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-token\") pod \"e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:44.864315 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:44.864282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:45.045911 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.045883 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5"] Jul 29 07:50:45.048563 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:50:45.048534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5a672b_e656_4d64_8dac_f04098b8a044.slice/crio-be02f71ff53c6682633e3481a5f5d6f300ecb49137f48a99bbe922d04fb8bc92 WatchSource:0}: Error finding container be02f71ff53c6682633e3481a5f5d6f300ecb49137f48a99bbe922d04fb8bc92: Status 404 returned error can't find the container with id be02f71ff53c6682633e3481a5f5d6f300ecb49137f48a99bbe922d04fb8bc92 Jul 29 07:50:45.050730 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.050700 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:50:45.050804 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.050765 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:50:45.050804 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.050794 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:50:45.510965 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.510930 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:45.711635 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.711596 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" event={"ID":"2d5a672b-e656-4d64-8dac-f04098b8a044","Type":"ContainerStarted","Data":"b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020"} Jul 29 07:50:45.712102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.711671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" event={"ID":"2d5a672b-e656-4d64-8dac-f04098b8a044","Type":"ContainerStarted","Data":"be02f71ff53c6682633e3481a5f5d6f300ecb49137f48a99bbe922d04fb8bc92"} Jul 29 07:50:45.771975 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.771859 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podStartSLOduration=1.771841717 podStartE2EDuration="1.771841717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:50:45.77169178 +0000 UTC m=+2155.543142578" watchObservedRunningTime="2026-07-29 07:50:45.771841717 +0000 UTC m=+2155.543292516" Jul 29 07:50:45.864827 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:45.864786 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:46.870702 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:46.870671 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:47.724102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:47.724060 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:47.725110 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:47.725083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:50:48.052842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:48.052738 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:48.451934 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:48.451899 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:48.782388 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:48.782305 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:49.913250 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:49.913210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:50.111590 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:50.111560 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:50:51.025941 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:51.025904 2568 scope.go:117] "RemoveContainer" containerID="366dc28a3967ac9dd98ba61735aee8b23e966c3ec47661a9f39a5ab0040644ac" Jul 29 07:50:51.035764 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:50:51.035733 2568 scope.go:117] "RemoveContainer" containerID="c4e9f2adbfd08068221a638e721a6a61279b5c33c9d44b669014232ef8d1a0ec" Jul 29 07:51:02.924721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:02.923244 2568 patch_prober.go:28] interesting pod/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": read tcp 10.132.0.2:36200->10.132.0.69:15021: read: connection reset by peer" start-of-body= Jul 29 07:51:02.924721 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:02.923352 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": read tcp 10.132.0.2:36200->10.132.0.69:15021: read: connection reset by peer" Jul 29 07:51:03.812325 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:03.812290 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerID="b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020" exitCode=0 Jul 29 07:51:03.812555 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:03.812364 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" event={"ID":"2d5a672b-e656-4d64-8dac-f04098b8a044","Type":"ContainerDied","Data":"b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020"} Jul 29 07:51:03.812782 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:03.812766 2568 scope.go:117] "RemoveContainer" containerID="b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020" Jul 29 07:51:03.813840 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:03.813814 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:03.813905 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:03.813884 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:03.813944 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:03.813918 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:04.819367 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:04.819325 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" event={"ID":"2d5a672b-e656-4d64-8dac-f04098b8a044","Type":"ContainerStarted","Data":"eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940"} Jul 29 07:51:04.864918 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:04.864876 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:51:04.865297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:04.865269 2568 patch_prober.go:28] interesting pod/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 07:51:04.865350 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:04.865321 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 07:51:05.865434 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:05.865396 2568 patch_prober.go:28] interesting pod/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 07:51:05.865856 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:05.865459 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 07:51:06.865339 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:06.865295 2568 patch_prober.go:28] interesting pod/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 07:51:06.865550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:06.865357 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 07:51:07.864998 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:07.864956 2568 patch_prober.go:28] interesting pod/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 07:51:07.865211 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:07.865018 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 07:51:08.865666 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:08.865620 2568 patch_prober.go:28] interesting pod/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 07:51:08.866112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:08.865683 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 07:51:09.868305 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:09.868267 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:51:09.868790 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:09.868659 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:51:09.869493 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:09.869472 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:51:12.978181 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:12.978119 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:13.382869 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.382769 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:13.705300 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.705249 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x"] Jul 29 07:51:13.710932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.710897 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf943cf-f690-4a50-902d-be206e9d903d" containerName="istio-proxy" Jul 29 07:51:13.715324 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.715294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.720443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.720407 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d6451b-openshift-default-dockercfg-crgml\"" Jul 29 07:51:13.732457 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.732414 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x"] Jul 29 07:51:13.756132 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.756099 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:13.800853 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.800817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-token\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.800877 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-credential-socket\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.800947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-data\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.801024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxvm\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-kube-api-access-4pxvm\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801098 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.801088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-envoy\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801359 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.801119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-podinfo\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801359 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.801151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-certs\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801359 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.801245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-socket\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.801359 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.801271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e413acc-785d-47e0-9e97-2fc260b7744f-istiod-ca-cert\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902329 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-podinfo\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902329 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-certs\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-socket\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e413acc-785d-47e0-9e97-2fc260b7744f-istiod-ca-cert\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-token\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-credential-socket\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-data\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxvm\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-kube-api-access-4pxvm\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-envoy\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902710 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-certs\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.902818 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-socket\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.903113 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.902947 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-data\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.903169 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.903113 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-credential-socket\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.903242 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.903221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e413acc-785d-47e0-9e97-2fc260b7744f-istiod-ca-cert\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.905137 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.905113 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-envoy\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.905391 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.905375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-podinfo\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.915762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.915732 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-token\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.916077 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.916036 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxvm\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-kube-api-access-4pxvm\") pod \"e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:13.991591 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:13.991497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:14.031947 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.031906 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:14.196781 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.196753 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x"] Jul 29 07:51:14.199139 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:51:14.199110 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e413acc_785d_47e0_9e97_2fc260b7744f.slice/crio-2d707a6ab0bf93ed49dd1c11a3c119cd341d97ce3bcfea85de2f4fe29711dfd4 WatchSource:0}: Error finding container 2d707a6ab0bf93ed49dd1c11a3c119cd341d97ce3bcfea85de2f4fe29711dfd4: Status 404 returned error can't find the container with id 2d707a6ab0bf93ed49dd1c11a3c119cd341d97ce3bcfea85de2f4fe29711dfd4 Jul 29 07:51:14.201695 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.201655 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:14.201795 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.201734 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:14.201795 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.201770 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:14.884778 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.884736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" event={"ID":"1e413acc-785d-47e0-9e97-2fc260b7744f","Type":"ContainerStarted","Data":"4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76"} Jul 29 07:51:14.884960 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.884786 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" event={"ID":"1e413acc-785d-47e0-9e97-2fc260b7744f","Type":"ContainerStarted","Data":"2d707a6ab0bf93ed49dd1c11a3c119cd341d97ce3bcfea85de2f4fe29711dfd4"} Jul 29 07:51:14.910712 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:14.910652 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podStartSLOduration=1.910634265 podStartE2EDuration="1.910634265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:51:14.909451318 +0000 UTC m=+2184.680902117" watchObservedRunningTime="2026-07-29 07:51:14.910634265 +0000 UTC m=+2184.682085064" Jul 29 07:51:15.032744 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:15.032709 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:15.038739 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:15.038711 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:15.890123 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:15.890080 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:15.891193 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:15.891170 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:17.013603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:17.013569 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:17.122898 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:17.122863 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:17.287372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:17.287290 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:17.406109 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:17.406076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:17.934619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:17.934588 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:18.091302 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:18.091266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:31.932268 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:31.932219 2568 patch_prober.go:28] interesting pod/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:51:31.932697 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:31.932296 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:51:33.998510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:33.998475 2568 generic.go:356] "Generic (PLEG): container finished" podID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerID="4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76" exitCode=0 Jul 29 07:51:33.999002 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:33.998525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" event={"ID":"1e413acc-785d-47e0-9e97-2fc260b7744f","Type":"ContainerDied","Data":"4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76"} Jul 29 07:51:33.999002 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:33.998952 2568 scope.go:117] "RemoveContainer" containerID="4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76" Jul 29 07:51:34.000089 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:34.000056 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:34.000201 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:34.000139 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:34.000201 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:34.000189 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:51:35.005777 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:35.005740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" event={"ID":"1e413acc-785d-47e0-9e97-2fc260b7744f","Type":"ContainerStarted","Data":"f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531"} Jul 29 07:51:35.034605 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:35.034564 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:35.035096 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:35.035064 2568 patch_prober.go:28] interesting pod/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 07:51:35.035218 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:35.035135 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 07:51:36.033349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:36.033307 2568 patch_prober.go:28] interesting pod/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 07:51:36.033996 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:36.033954 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 07:51:37.032806 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:37.032764 2568 patch_prober.go:28] interesting pod/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 07:51:37.033145 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:37.033114 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 07:51:37.410893 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:37.410864 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:51:38.033377 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:38.033334 2568 patch_prober.go:28] interesting pod/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 07:51:38.033696 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:38.033665 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 07:51:39.040247 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:39.040217 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:39.040685 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:39.040521 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:39.041185 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:39.041169 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:51:47.156773 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:51:47.156736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:52:03.260246 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:52:03.260210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:52:43.764400 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:52:43.764360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:52:44.278367 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:52:44.278332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:23.104261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:23.104224 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:24.126564 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.126506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:24.242295 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.242253 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl"] Jul 29 07:53:24.246557 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.246524 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:24.251882 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.251853 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:53:24.281710 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.281672 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl"] Jul 29 07:53:24.338403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.338356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa203c-9a13-41d9-b823-05af40f77980-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl\" (UID: \"d0fa203c-9a13-41d9-b823-05af40f77980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:24.439965 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.439930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa203c-9a13-41d9-b823-05af40f77980-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl\" (UID: \"d0fa203c-9a13-41d9-b823-05af40f77980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:24.440550 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.440530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa203c-9a13-41d9-b823-05af40f77980-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl\" (UID: \"d0fa203c-9a13-41d9-b823-05af40f77980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:24.558195 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.558136 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:24.930306 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:24.930275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl"] Jul 29 07:53:24.933401 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:53:24.933372 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fa203c_9a13_41d9_b823_05af40f77980.slice/crio-ca5603b47350b2bd37237343f73d57159c5fc4cfcd10d24972423cdea0ca3348 WatchSource:0}: Error finding container ca5603b47350b2bd37237343f73d57159c5fc4cfcd10d24972423cdea0ca3348: Status 404 returned error can't find the container with id ca5603b47350b2bd37237343f73d57159c5fc4cfcd10d24972423cdea0ca3348 Jul 29 07:53:25.620684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:25.620572 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" event={"ID":"d0fa203c-9a13-41d9-b823-05af40f77980","Type":"ContainerStarted","Data":"7a4b19da5a13020861ddf77c3eb076364d94411f62b8a046abf23958bdb9d602"} Jul 29 07:53:25.620684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:25.620642 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" event={"ID":"d0fa203c-9a13-41d9-b823-05af40f77980","Type":"ContainerStarted","Data":"ca5603b47350b2bd37237343f73d57159c5fc4cfcd10d24972423cdea0ca3348"} Jul 29 07:53:26.628541 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:26.628505 2568 generic.go:356] "Generic (PLEG): container finished" podID="d0fa203c-9a13-41d9-b823-05af40f77980" containerID="7a4b19da5a13020861ddf77c3eb076364d94411f62b8a046abf23958bdb9d602" exitCode=0 Jul 29 07:53:26.628953 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:26.628586 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" event={"ID":"d0fa203c-9a13-41d9-b823-05af40f77980","Type":"ContainerDied","Data":"7a4b19da5a13020861ddf77c3eb076364d94411f62b8a046abf23958bdb9d602"} Jul 29 07:53:27.784715 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:27.784683 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:27.872464 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:27.872425 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa203c-9a13-41d9-b823-05af40f77980-maas-api-service-ca\") pod \"d0fa203c-9a13-41d9-b823-05af40f77980\" (UID: \"d0fa203c-9a13-41d9-b823-05af40f77980\") " Jul 29 07:53:27.872840 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:27.872813 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fa203c-9a13-41d9-b823-05af40f77980-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0fa203c-9a13-41d9-b823-05af40f77980" (UID: "d0fa203c-9a13-41d9-b823-05af40f77980"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:27.973889 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:27.973847 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa203c-9a13-41d9-b823-05af40f77980-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:28.640581 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:28.640547 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" Jul 29 07:53:28.640768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:28.640541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl" event={"ID":"d0fa203c-9a13-41d9-b823-05af40f77980","Type":"ContainerDied","Data":"ca5603b47350b2bd37237343f73d57159c5fc4cfcd10d24972423cdea0ca3348"} Jul 29 07:53:28.640768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:28.640680 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5603b47350b2bd37237343f73d57159c5fc4cfcd10d24972423cdea0ca3348" Jul 29 07:53:35.052940 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:35.052884 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:35.165286 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:35.165253 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:35.301846 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:35.301807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:35.309227 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:35.309134 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl"] Jul 29 07:53:35.314632 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:35.314597 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d6491-4krwl"] Jul 29 07:53:35.705189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:35.705153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:36.743185 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:36.740901 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fa203c-9a13-41d9-b823-05af40f77980" path="/var/lib/kubelet/pods/d0fa203c-9a13-41d9-b823-05af40f77980/volumes" Jul 29 07:53:42.727653 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:42.727563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:42.828653 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:42.828609 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:43.045350 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.045244 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5"] Jul 29 07:53:43.045548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.045519 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" containerID="cri-o://eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940" gracePeriod=30 Jul 29 07:53:43.100609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.100571 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:43.313492 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.313398 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:43.530440 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.530400 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9"] Jul 29 07:53:43.530923 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.530907 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fa203c-9a13-41d9-b823-05af40f77980" containerName="revoke-keys" Jul 29 07:53:43.530970 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.530928 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fa203c-9a13-41d9-b823-05af40f77980" containerName="revoke-keys" Jul 29 07:53:43.531005 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.530996 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fa203c-9a13-41d9-b823-05af40f77980" containerName="revoke-keys" Jul 29 07:53:43.534493 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.534473 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:43.539186 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.539164 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:53:43.558254 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.558224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9"] Jul 29 07:53:43.623400 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.623252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd99be13-6726-4219-ba55-e87462378db0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9\" (UID: \"dd99be13-6726-4219-ba55-e87462378db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:43.724667 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.724635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd99be13-6726-4219-ba55-e87462378db0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9\" (UID: \"dd99be13-6726-4219-ba55-e87462378db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:43.725331 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.725310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd99be13-6726-4219-ba55-e87462378db0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9\" (UID: \"dd99be13-6726-4219-ba55-e87462378db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:43.846366 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:43.846322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:44.207289 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:44.207257 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9"] Jul 29 07:53:44.209385 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:53:44.209359 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd99be13_6726_4219_ba55_e87462378db0.slice/crio-637a1c911969f63a4f791effa1fb76fbf1c968d2d459a0225ff68dbb23647aeb WatchSource:0}: Error finding container 637a1c911969f63a4f791effa1fb76fbf1c968d2d459a0225ff68dbb23647aeb: Status 404 returned error can't find the container with id 637a1c911969f63a4f791effa1fb76fbf1c968d2d459a0225ff68dbb23647aeb Jul 29 07:53:44.730060 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:44.729998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" event={"ID":"dd99be13-6726-4219-ba55-e87462378db0","Type":"ContainerStarted","Data":"aa79cfb46799dd90b79db746b2802cc049020022f83123fcd260e60129f0bd85"} Jul 29 07:53:44.730238 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:44.730095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" event={"ID":"dd99be13-6726-4219-ba55-e87462378db0","Type":"ContainerStarted","Data":"637a1c911969f63a4f791effa1fb76fbf1c968d2d459a0225ff68dbb23647aeb"} Jul 29 07:53:44.757859 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:44.757806 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" podStartSLOduration=1.757788341 podStartE2EDuration="1.757788341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:53:44.754860026 +0000 UTC m=+2334.526310826" watchObservedRunningTime="2026-07-29 07:53:44.757788341 +0000 UTC m=+2334.529239139" Jul 29 07:53:45.737772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:45.737737 2568 generic.go:356] "Generic (PLEG): container finished" podID="dd99be13-6726-4219-ba55-e87462378db0" containerID="aa79cfb46799dd90b79db746b2802cc049020022f83123fcd260e60129f0bd85" exitCode=0 Jul 29 07:53:45.737772 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:45.737776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" event={"ID":"dd99be13-6726-4219-ba55-e87462378db0","Type":"ContainerDied","Data":"aa79cfb46799dd90b79db746b2802cc049020022f83123fcd260e60129f0bd85"} Jul 29 07:53:46.891248 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:46.891221 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:47.054387 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:47.054279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd99be13-6726-4219-ba55-e87462378db0-maas-api-service-ca\") pod \"dd99be13-6726-4219-ba55-e87462378db0\" (UID: \"dd99be13-6726-4219-ba55-e87462378db0\") " Jul 29 07:53:47.054657 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:47.054634 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd99be13-6726-4219-ba55-e87462378db0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd99be13-6726-4219-ba55-e87462378db0" (UID: "dd99be13-6726-4219-ba55-e87462378db0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:47.156007 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:47.155963 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd99be13-6726-4219-ba55-e87462378db0-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:47.750200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:47.750170 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" Jul 29 07:53:47.750200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:47.750196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9" event={"ID":"dd99be13-6726-4219-ba55-e87462378db0","Type":"ContainerDied","Data":"637a1c911969f63a4f791effa1fb76fbf1c968d2d459a0225ff68dbb23647aeb"} Jul 29 07:53:47.750418 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:47.750242 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637a1c911969f63a4f791effa1fb76fbf1c968d2d459a0225ff68dbb23647aeb" Jul 29 07:53:48.321791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.321699 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:53:48.467524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467481 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-socket\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467524 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467525 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-token\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467564 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-envoy\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467587 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-data\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467620 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-credential-socket\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467679 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a672b-e656-4d64-8dac-f04098b8a044-istiod-ca-cert\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467766 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qbvv\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-kube-api-access-5qbvv\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.467798 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467797 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-certs\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.468136 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467840 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-podinfo\") pod \"2d5a672b-e656-4d64-8dac-f04098b8a044\" (UID: \"2d5a672b-e656-4d64-8dac-f04098b8a044\") " Jul 29 07:53:48.468196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.467839 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:53:48.468196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.468036 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:53:48.468196 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.468108 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5a672b-e656-4d64-8dac-f04098b8a044-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:48.468337 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.468289 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:53:48.468883 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.468856 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:53:48.470302 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.470264 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-kube-api-access-5qbvv" (OuterVolumeSpecName: "kube-api-access-5qbvv") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "kube-api-access-5qbvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:48.470420 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.470316 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:48.470726 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.470704 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:53:48.470830 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.470719 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d5a672b-e656-4d64-8dac-f04098b8a044" (UID: "2d5a672b-e656-4d64-8dac-f04098b8a044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:53:48.569552 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569506 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569552 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569542 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569552 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569556 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569568 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a672b-e656-4d64-8dac-f04098b8a044-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569580 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbvv\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-kube-api-access-5qbvv\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569591 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569602 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569612 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a672b-e656-4d64-8dac-f04098b8a044-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.569823 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.569623 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a672b-e656-4d64-8dac-f04098b8a044-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:53:48.757789 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.757757 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerID="eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940" exitCode=0 Jul 29 07:53:48.757973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.757803 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" Jul 29 07:53:48.757973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.757801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" event={"ID":"2d5a672b-e656-4d64-8dac-f04098b8a044","Type":"ContainerDied","Data":"eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940"} Jul 29 07:53:48.757973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.757935 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5" event={"ID":"2d5a672b-e656-4d64-8dac-f04098b8a044","Type":"ContainerDied","Data":"be02f71ff53c6682633e3481a5f5d6f300ecb49137f48a99bbe922d04fb8bc92"} Jul 29 07:53:48.757973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.757970 2568 scope.go:117] "RemoveContainer" containerID="eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940" Jul 29 07:53:48.768306 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.768285 2568 scope.go:117] "RemoveContainer" containerID="b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020" Jul 29 07:53:48.777904 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.777881 2568 scope.go:117] "RemoveContainer" containerID="eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940" Jul 29 07:53:48.778283 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:53:48.778264 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940\": container with ID starting with eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940 not found: ID does not exist" containerID="eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940" Jul 29 07:53:48.778354 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.778293 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940"} err="failed to get container status \"eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940\": rpc error: code = NotFound desc = could not find container \"eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940\": container with ID starting with eaf7e6ca987e9b4c4ca5e5c863253922d6a27ffb4700c085846eae62547c4940 not found: ID does not exist" Jul 29 07:53:48.778354 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.778315 2568 scope.go:117] "RemoveContainer" containerID="b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020" Jul 29 07:53:48.778637 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:53:48.778617 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020\": container with ID starting with b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020 not found: ID does not exist" containerID="b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020" Jul 29 07:53:48.778688 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.778645 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020"} err="failed to get container status \"b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020\": rpc error: code = NotFound desc = could not find container \"b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020\": container with ID starting with b177d3088295e5891c1050283de1e44664a69b2b9bb5951187d0b1d477abc020 not found: ID does not exist" Jul 29 07:53:48.795203 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.795161 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5"] Jul 29 07:53:48.805165 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:48.805130 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7d6491-openshift-default-c45674bd8-9qtp5"] Jul 29 07:53:50.739200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:50.739163 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" path="/var/lib/kubelet/pods/2d5a672b-e656-4d64-8dac-f04098b8a044/volumes" Jul 29 07:53:51.079021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:51.078939 2568 scope.go:117] "RemoveContainer" containerID="ba6c2f6beeeb99353fd0d938e1995157e49909ac277d7baf6e6aae41204a2654" Jul 29 07:53:54.513113 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:54.513078 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:54.665539 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:54.665502 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:54.761594 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:54.761559 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:54.835398 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:54.835294 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9"] Jul 29 07:53:54.844088 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:54.844027 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d6451b-kv8h9"] Jul 29 07:53:54.852097 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:54.852067 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:55.269384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:55.269353 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:53:56.738774 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:53:56.738743 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd99be13-6726-4219-ba55-e87462378db0" path="/var/lib/kubelet/pods/dd99be13-6726-4219-ba55-e87462378db0/volumes" Jul 29 07:54:02.204415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:02.204379 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:02.293762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:02.293728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:02.454384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:02.454340 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x"] Jul 29 07:54:02.454692 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:02.454601 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" containerID="cri-o://f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531" gracePeriod=30 Jul 29 07:54:02.485647 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:02.485611 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:02.848609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:02.848512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:03.079226 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.079190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:03.427284 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427248 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx"] Jul 29 07:54:03.427768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427691 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" Jul 29 07:54:03.427768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427706 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" Jul 29 07:54:03.427768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427726 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd99be13-6726-4219-ba55-e87462378db0" containerName="revoke-keys" Jul 29 07:54:03.427768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427732 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd99be13-6726-4219-ba55-e87462378db0" containerName="revoke-keys" Jul 29 07:54:03.427768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427746 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" Jul 29 07:54:03.427768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427752 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" Jul 29 07:54:03.428084 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427827 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" Jul 29 07:54:03.428084 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427836 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5a672b-e656-4d64-8dac-f04098b8a044" containerName="istio-proxy" Jul 29 07:54:03.428084 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.427842 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd99be13-6726-4219-ba55-e87462378db0" containerName="revoke-keys" Jul 29 07:54:03.431470 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.431443 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.438443 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.438418 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0d455e-openshift-default-dockercfg-4s77b\"" Jul 29 07:54:03.457812 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.457765 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx"] Jul 29 07:54:03.467836 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.467807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:03.509759 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.509724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-token\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.509948 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.509765 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-certs\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.509948 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.509791 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-envoy\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.509948 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.509880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-data\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.509948 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.509933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-credential-socket\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.510135 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.509979 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb931d56-ed85-4511-8a16-77785f6e64b7-istiod-ca-cert\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.510135 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.510016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-socket\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.510135 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.510109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-podinfo\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.510234 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.510164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9q4z\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-kube-api-access-v9q4z\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.610961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.610920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-data\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.610961 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.610970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-credential-socket\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.610991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb931d56-ed85-4511-8a16-77785f6e64b7-istiod-ca-cert\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-socket\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-podinfo\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9q4z\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-kube-api-access-v9q4z\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-token\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-certs\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611251 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-envoy\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611626 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611448 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-credential-socket\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611684 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-certs\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611746 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611704 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-socket\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611806 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-data\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.611907 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.611883 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb931d56-ed85-4511-8a16-77785f6e64b7-istiod-ca-cert\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.613719 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.613694 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-envoy\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.613862 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.613843 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-podinfo\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.624079 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.620540 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-token\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.624079 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.620809 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9q4z\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-kube-api-access-v9q4z\") pod \"e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.743201 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.743106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:03.926529 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.926501 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx"] Jul 29 07:54:03.929481 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:54:03.929450 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb931d56_ed85_4511_8a16_77785f6e64b7.slice/crio-2935b9d3c787f911cc5f3486f6d22f952fd95ec807cdfa0693159a2800328d0a WatchSource:0}: Error finding container 2935b9d3c787f911cc5f3486f6d22f952fd95ec807cdfa0693159a2800328d0a: Status 404 returned error can't find the container with id 2935b9d3c787f911cc5f3486f6d22f952fd95ec807cdfa0693159a2800328d0a Jul 29 07:54:03.932154 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.932112 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:03.932234 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.932207 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:03.932278 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.932257 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:03.938570 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:03.938526 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:04.860116 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:04.860071 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" event={"ID":"cb931d56-ed85-4511-8a16-77785f6e64b7","Type":"ContainerStarted","Data":"9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443"} Jul 29 07:54:04.860530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:04.860138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" event={"ID":"cb931d56-ed85-4511-8a16-77785f6e64b7","Type":"ContainerStarted","Data":"2935b9d3c787f911cc5f3486f6d22f952fd95ec807cdfa0693159a2800328d0a"} Jul 29 07:54:05.744229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:05.744184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:05.750012 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:05.749980 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:05.773412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:05.773353 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podStartSLOduration=2.773336211 podStartE2EDuration="2.773336211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:04.893761613 +0000 UTC m=+2354.665212411" watchObservedRunningTime="2026-07-29 07:54:05.773336211 +0000 UTC m=+2355.544787009" Jul 29 07:54:05.866372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:05.866333 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:05.867375 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:05.867356 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:07.090116 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.090081 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:07.324261 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.324222 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:07.478228 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.478185 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:07.729993 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.729923 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:54:07.851397 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851359 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-token\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851397 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851394 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-podinfo\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851414 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-envoy\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851436 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pxvm\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-kube-api-access-4pxvm\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851480 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e413acc-785d-47e0-9e97-2fc260b7744f-istiod-ca-cert\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851500 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-data\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851538 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-socket\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851621 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-credential-socket\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.851799 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851715 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-certs\") pod \"1e413acc-785d-47e0-9e97-2fc260b7744f\" (UID: \"1e413acc-785d-47e0-9e97-2fc260b7744f\") " Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851844 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.851938 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e413acc-785d-47e0-9e97-2fc260b7744f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.852069 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.852183 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.852247 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e413acc-785d-47e0-9e97-2fc260b7744f-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.852302 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.852362 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:54:07.852423 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.852382 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.854033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.854005 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:54:07.854033 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.854024 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-kube-api-access-4pxvm" (OuterVolumeSpecName: "kube-api-access-4pxvm") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "kube-api-access-4pxvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:54:07.854229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.854011 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:54:07.854229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.854199 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e413acc-785d-47e0-9e97-2fc260b7744f" (UID: "1e413acc-785d-47e0-9e97-2fc260b7744f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:54:07.879862 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.879829 2568 generic.go:356] "Generic (PLEG): container finished" podID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerID="f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531" exitCode=0 Jul 29 07:54:07.880060 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.879881 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" Jul 29 07:54:07.880060 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.879918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" event={"ID":"1e413acc-785d-47e0-9e97-2fc260b7744f","Type":"ContainerDied","Data":"f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531"} Jul 29 07:54:07.880060 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.879977 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x" event={"ID":"1e413acc-785d-47e0-9e97-2fc260b7744f","Type":"ContainerDied","Data":"2d707a6ab0bf93ed49dd1c11a3c119cd341d97ce3bcfea85de2f4fe29711dfd4"} Jul 29 07:54:07.880060 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.880008 2568 scope.go:117] "RemoveContainer" containerID="f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531" Jul 29 07:54:07.890984 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.890964 2568 scope.go:117] "RemoveContainer" containerID="4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76" Jul 29 07:54:07.900797 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.900773 2568 scope.go:117] "RemoveContainer" containerID="f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531" Jul 29 07:54:07.901181 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:54:07.901160 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531\": container with ID starting with f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531 not found: ID does not exist" containerID="f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531" Jul 29 07:54:07.901252 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.901193 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531"} err="failed to get container status \"f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531\": rpc error: code = NotFound desc = could not find container \"f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531\": container with ID starting with f8d1a3c4f1ded9967d9548531a8ed28f94dbb8eb3aa2fdf27c214cb93ddfe531 not found: ID does not exist" Jul 29 07:54:07.901252 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.901216 2568 scope.go:117] "RemoveContainer" containerID="4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76" Jul 29 07:54:07.901486 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:54:07.901468 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76\": container with ID starting with 4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76 not found: ID does not exist" containerID="4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76" Jul 29 07:54:07.901530 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.901493 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76"} err="failed to get container status \"4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76\": rpc error: code = NotFound desc = could not find container \"4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76\": container with ID starting with 4cbf4f407a8ddcea28c4ec4906b140e275ce836352c3f0d3f06c395dafc63e76 not found: ID does not exist" Jul 29 07:54:07.910431 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.910391 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x"] Jul 29 07:54:07.914452 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.914417 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d6451b-openshift-default-6d7c96695b-rw99x"] Jul 29 07:54:07.953598 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.953558 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.953598 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.953595 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.953598 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.953608 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.953842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.953620 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pxvm\" (UniqueName: \"kubernetes.io/projected/1e413acc-785d-47e0-9e97-2fc260b7744f-kube-api-access-4pxvm\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.953842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.953632 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:07.953842 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:07.953645 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e413acc-785d-47e0-9e97-2fc260b7744f-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:54:08.172054 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:08.172018 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:08.268541 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:08.268498 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:08.738468 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:08.738431 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" path="/var/lib/kubelet/pods/1e413acc-785d-47e0-9e97-2fc260b7744f/volumes" Jul 29 07:54:21.908241 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:21.908194 2568 patch_prober.go:28] interesting pod/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:54:21.908654 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:21.908257 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:54:22.977181 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:22.977143 2568 generic.go:356] "Generic (PLEG): container finished" podID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerID="9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443" exitCode=0 Jul 29 07:54:22.977634 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:22.977224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" event={"ID":"cb931d56-ed85-4511-8a16-77785f6e64b7","Type":"ContainerDied","Data":"9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443"} Jul 29 07:54:22.977713 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:22.977694 2568 scope.go:117] "RemoveContainer" containerID="9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443" Jul 29 07:54:22.978881 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:22.978850 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:22.978984 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:22.978936 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:22.979074 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:22.978997 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:23.984488 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:23.984449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" event={"ID":"cb931d56-ed85-4511-8a16-77785f6e64b7","Type":"ContainerStarted","Data":"187c720fea61a63975321a50fd0cadf70ffa53299ded676b81c68df4df576926"} Jul 29 07:54:24.743737 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:24.743700 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:24.744145 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:24.744113 2568 patch_prober.go:28] interesting pod/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 29 07:54:24.744269 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:24.744176 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 29 07:54:25.743718 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:25.743675 2568 patch_prober.go:28] interesting pod/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 29 07:54:25.744476 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:25.744439 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 29 07:54:26.744592 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:26.744555 2568 patch_prober.go:28] interesting pod/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 29 07:54:26.745178 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:26.745139 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 29 07:54:27.744202 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:27.744148 2568 patch_prober.go:28] interesting pod/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 29 07:54:27.744440 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:27.744225 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 29 07:54:28.747768 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:28.747735 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:28.748221 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:28.747992 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:28.748851 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:28.748827 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:54:32.019904 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.019866 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:32.549227 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.549194 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:32.855835 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.855745 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv"] Jul 29 07:54:32.856485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.856467 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" Jul 29 07:54:32.856548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.856489 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" Jul 29 07:54:32.856548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.856504 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" Jul 29 07:54:32.856548 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.856513 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" Jul 29 07:54:32.856647 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.856599 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" Jul 29 07:54:32.856647 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.856612 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e413acc-785d-47e0-9e97-2fc260b7744f" containerName="istio-proxy" Jul 29 07:54:32.860192 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.860170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.863991 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.863963 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fd8a8c-openshift-default-dockercfg-dpxjw\"" Jul 29 07:54:32.872889 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.872863 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv"] Jul 29 07:54:32.892104 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.892064 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:32.998619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998571 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-istio-token\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.998619 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjzn\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-kube-api-access-9gjzn\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-envoy\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8435df18-f92c-404d-a80d-6c8e1f586043-istio-podinfo\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-socket\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998843 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-credential-socket\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998891 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-data\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-certs\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:32.999133 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:32.998961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8435df18-f92c-404d-a80d-6c8e1f586043-istiod-ca-cert\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.070332 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.070289 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:33.099510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099470 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-envoy\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099510 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8435df18-f92c-404d-a80d-6c8e1f586043-istio-podinfo\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099533 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-socket\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099558 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-credential-socket\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099722 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-data\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099890 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-certs\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099890 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8435df18-f92c-404d-a80d-6c8e1f586043-istiod-ca-cert\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099890 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099844 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-istio-token\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.099890 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099873 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjzn\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-kube-api-access-9gjzn\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.100129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099961 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-credential-socket\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.100129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.099992 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-socket\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.100307 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.100282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-data\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.100366 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.100314 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-certs\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.100811 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.100786 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8435df18-f92c-404d-a80d-6c8e1f586043-istiod-ca-cert\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.102254 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.102224 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-envoy\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.102395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.102361 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8435df18-f92c-404d-a80d-6c8e1f586043-istio-podinfo\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.111426 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.111339 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-istio-token\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.111574 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.111532 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjzn\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-kube-api-access-9gjzn\") pod \"e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.172250 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.172206 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:33.337791 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.337764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv"] Jul 29 07:54:33.339775 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:54:33.339744 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8435df18_f92c_404d_a80d_6c8e1f586043.slice/crio-182879e181ee9e70d983160dad2e319a910b63448ee6af3837d991dc29e78457 WatchSource:0}: Error finding container 182879e181ee9e70d983160dad2e319a910b63448ee6af3837d991dc29e78457: Status 404 returned error can't find the container with id 182879e181ee9e70d983160dad2e319a910b63448ee6af3837d991dc29e78457 Jul 29 07:54:33.341835 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.341798 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:33.341940 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.341869 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:33.341940 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:33.341928 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:34.040199 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:34.040166 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" event={"ID":"8435df18-f92c-404d-a80d-6c8e1f586043","Type":"ContainerStarted","Data":"41a3742554c82a0b1b3cc255315113b1c88b9053bcfec492407fb81858a18528"} Jul 29 07:54:34.040382 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:34.040219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" event={"ID":"8435df18-f92c-404d-a80d-6c8e1f586043","Type":"ContainerStarted","Data":"182879e181ee9e70d983160dad2e319a910b63448ee6af3837d991dc29e78457"} Jul 29 07:54:34.072117 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:34.072057 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podStartSLOduration=2.072015142 podStartE2EDuration="2.072015142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:34.069648768 +0000 UTC m=+2383.841099563" watchObservedRunningTime="2026-07-29 07:54:34.072015142 +0000 UTC m=+2383.843465938" Jul 29 07:54:34.172832 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:34.172783 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:34.177652 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:34.177621 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:35.046144 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:35.046106 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:35.047302 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:35.047279 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:36.115575 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:36.115531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:36.510437 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:36.510401 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:36.610438 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:36.610403 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:36.840380 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:36.840274 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:36.933332 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:36.933289 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:54:49.138194 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:49.138155 2568 generic.go:356] "Generic (PLEG): container finished" podID="8435df18-f92c-404d-a80d-6c8e1f586043" containerID="41a3742554c82a0b1b3cc255315113b1c88b9053bcfec492407fb81858a18528" exitCode=0 Jul 29 07:54:49.138624 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:49.138226 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" event={"ID":"8435df18-f92c-404d-a80d-6c8e1f586043","Type":"ContainerDied","Data":"41a3742554c82a0b1b3cc255315113b1c88b9053bcfec492407fb81858a18528"} Jul 29 07:54:49.138744 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:49.138724 2568 scope.go:117] "RemoveContainer" containerID="41a3742554c82a0b1b3cc255315113b1c88b9053bcfec492407fb81858a18528" Jul 29 07:54:49.140094 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:49.140057 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:49.140219 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:49.140150 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:49.140219 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:49.140203 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:54:50.046637 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.046533 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:50.149787 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.149738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" event={"ID":"8435df18-f92c-404d-a80d-6c8e1f586043","Type":"ContainerStarted","Data":"a3cb5d41565fcf672ed37a538e705dbbc9b3923125032c715e5e07969a723fac"} Jul 29 07:54:50.172407 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.172370 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:50.173558 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.173527 2568 patch_prober.go:28] interesting pod/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 29 07:54:50.173694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.173586 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 29 07:54:50.859913 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.859881 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:54:50.869873 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:50.869845 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:54:51.106419 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:51.106386 2568 scope.go:117] "RemoveContainer" containerID="f1a7ce84aa60104f9effb88cc50d9c916f820c7e699f1594ab3c76950ecb7b43" Jul 29 07:54:51.173204 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:51.173168 2568 patch_prober.go:28] interesting pod/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 29 07:54:51.173608 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:51.173224 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 29 07:54:52.172741 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:52.172697 2568 patch_prober.go:28] interesting pod/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 29 07:54:52.172940 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:52.172769 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 29 07:54:53.172778 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:53.172736 2568 patch_prober.go:28] interesting pod/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 29 07:54:53.173205 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:53.172838 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 29 07:54:54.177178 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:54.177135 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:54.177635 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:54.177547 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:54:54.178602 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:54:54.178580 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:55:01.194226 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:01.194187 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:04.423647 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.423600 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr"] Jul 29 07:55:04.431788 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.431752 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:04.440108 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.440075 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:55:04.440662 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.440630 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr"] Jul 29 07:55:04.498941 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.498898 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2387c431-9c59-494a-ba96-1426f438f489-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr\" (UID: \"2387c431-9c59-494a-ba96-1426f438f489\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:04.600502 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.600456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2387c431-9c59-494a-ba96-1426f438f489-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr\" (UID: \"2387c431-9c59-494a-ba96-1426f438f489\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:04.601156 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.601133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2387c431-9c59-494a-ba96-1426f438f489-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr\" (UID: \"2387c431-9c59-494a-ba96-1426f438f489\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:04.749501 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.749407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:04.918187 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:04.918153 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr"] Jul 29 07:55:04.921806 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:55:04.921771 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2387c431_9c59_494a_ba96_1426f438f489.slice/crio-2f898e0b69f2123df2c8b185de0470c953cf467c839b89ba11d73273d5f471e5 WatchSource:0}: Error finding container 2f898e0b69f2123df2c8b185de0470c953cf467c839b89ba11d73273d5f471e5: Status 404 returned error can't find the container with id 2f898e0b69f2123df2c8b185de0470c953cf467c839b89ba11d73273d5f471e5 Jul 29 07:55:05.248943 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:05.248902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" event={"ID":"2387c431-9c59-494a-ba96-1426f438f489","Type":"ContainerStarted","Data":"1f6a698a00fa49503fc4be437af02c1f24f31ce9f1ccf88860b7be352998e376"} Jul 29 07:55:05.249198 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:05.248962 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" event={"ID":"2387c431-9c59-494a-ba96-1426f438f489","Type":"ContainerStarted","Data":"2f898e0b69f2123df2c8b185de0470c953cf467c839b89ba11d73273d5f471e5"} Jul 29 07:55:05.265877 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:05.265818 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" podStartSLOduration=1.265798781 podStartE2EDuration="1.265798781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:55:05.264963689 +0000 UTC m=+2415.036414488" watchObservedRunningTime="2026-07-29 07:55:05.265798781 +0000 UTC m=+2415.037249580" Jul 29 07:55:06.256762 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:06.256721 2568 generic.go:356] "Generic (PLEG): container finished" podID="2387c431-9c59-494a-ba96-1426f438f489" containerID="1f6a698a00fa49503fc4be437af02c1f24f31ce9f1ccf88860b7be352998e376" exitCode=0 Jul 29 07:55:06.257220 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:06.256813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" event={"ID":"2387c431-9c59-494a-ba96-1426f438f489","Type":"ContainerDied","Data":"1f6a698a00fa49503fc4be437af02c1f24f31ce9f1ccf88860b7be352998e376"} Jul 29 07:55:07.416200 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:07.416171 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:07.530114 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:07.530004 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2387c431-9c59-494a-ba96-1426f438f489-maas-api-service-ca\") pod \"2387c431-9c59-494a-ba96-1426f438f489\" (UID: \"2387c431-9c59-494a-ba96-1426f438f489\") " Jul 29 07:55:07.530395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:07.530363 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2387c431-9c59-494a-ba96-1426f438f489-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2387c431-9c59-494a-ba96-1426f438f489" (UID: "2387c431-9c59-494a-ba96-1426f438f489"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:07.631559 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:07.631521 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2387c431-9c59-494a-ba96-1426f438f489-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:08.269011 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:08.268981 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" Jul 29 07:55:08.269011 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:08.269007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr" event={"ID":"2387c431-9c59-494a-ba96-1426f438f489","Type":"ContainerDied","Data":"2f898e0b69f2123df2c8b185de0470c953cf467c839b89ba11d73273d5f471e5"} Jul 29 07:55:08.269252 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:08.269090 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f898e0b69f2123df2c8b185de0470c953cf467c839b89ba11d73273d5f471e5" Jul 29 07:55:19.933350 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:19.933237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:20.004537 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.004499 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:20.068715 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.068676 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:20.110554 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.110509 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr"] Jul 29 07:55:20.113528 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.113496 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d455e-4f9nr"] Jul 29 07:55:20.176950 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.176902 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:20.531897 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.531856 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:20.738938 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:20.738903 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2387c431-9c59-494a-ba96-1426f438f489" path="/var/lib/kubelet/pods/2387c431-9c59-494a-ba96-1426f438f489/volumes" Jul 29 07:55:26.854117 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:26.854085 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:26.979508 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:26.979458 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:27.150007 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.149907 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx"] Jul 29 07:55:27.150189 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.150155 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" containerID="cri-o://187c720fea61a63975321a50fd0cadf70ffa53299ded676b81c68df4df576926" gracePeriod=30 Jul 29 07:55:27.167170 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.167132 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:27.432597 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.432543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:27.562648 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.562609 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn"] Jul 29 07:55:27.563081 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.563067 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2387c431-9c59-494a-ba96-1426f438f489" containerName="revoke-keys" Jul 29 07:55:27.563081 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.563081 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2387c431-9c59-494a-ba96-1426f438f489" containerName="revoke-keys" Jul 29 07:55:27.563169 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.563143 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2387c431-9c59-494a-ba96-1426f438f489" containerName="revoke-keys" Jul 29 07:55:27.566262 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.566241 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:27.571577 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.571555 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 07:55:27.585229 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.585185 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn"] Jul 29 07:55:27.628112 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.628067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00384ee-0fc8-461b-be48-0f1d182838ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn\" (UID: \"b00384ee-0fc8-461b-be48-0f1d182838ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:27.729694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.729596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00384ee-0fc8-461b-be48-0f1d182838ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn\" (UID: \"b00384ee-0fc8-461b-be48-0f1d182838ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:27.730441 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.730419 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00384ee-0fc8-461b-be48-0f1d182838ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn\" (UID: \"b00384ee-0fc8-461b-be48-0f1d182838ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:27.876921 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:27.876879 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:28.043352 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:28.043321 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn"] Jul 29 07:55:28.045131 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:55:28.045106 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00384ee_0fc8_461b_be48_0f1d182838ec.slice/crio-895cd758c7ffd3777dc2c86037b6e2b136e5208b20914101e6bf5f938079e430 WatchSource:0}: Error finding container 895cd758c7ffd3777dc2c86037b6e2b136e5208b20914101e6bf5f938079e430: Status 404 returned error can't find the container with id 895cd758c7ffd3777dc2c86037b6e2b136e5208b20914101e6bf5f938079e430 Jul 29 07:55:28.391773 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:28.391731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" event={"ID":"b00384ee-0fc8-461b-be48-0f1d182838ec","Type":"ContainerStarted","Data":"8dd5050c604cc20c3e6133a11ac523fbb961c4a09793b5c7e4de7361a308d93b"} Jul 29 07:55:28.392193 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:28.391837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" event={"ID":"b00384ee-0fc8-461b-be48-0f1d182838ec","Type":"ContainerStarted","Data":"895cd758c7ffd3777dc2c86037b6e2b136e5208b20914101e6bf5f938079e430"} Jul 29 07:55:28.422646 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:28.422586 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" podStartSLOduration=1.42256922 podStartE2EDuration="1.42256922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:55:28.420382396 +0000 UTC m=+2438.191833195" watchObservedRunningTime="2026-07-29 07:55:28.42256922 +0000 UTC m=+2438.194020018" Jul 29 07:55:28.750731 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:28.750694 2568 patch_prober.go:28] interesting pod/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:55:28.750912 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:28.750757 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:55:29.399395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:29.399298 2568 generic.go:356] "Generic (PLEG): container finished" podID="b00384ee-0fc8-461b-be48-0f1d182838ec" containerID="8dd5050c604cc20c3e6133a11ac523fbb961c4a09793b5c7e4de7361a308d93b" exitCode=0 Jul 29 07:55:29.399395 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:29.399349 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" event={"ID":"b00384ee-0fc8-461b-be48-0f1d182838ec","Type":"ContainerDied","Data":"8dd5050c604cc20c3e6133a11ac523fbb961c4a09793b5c7e4de7361a308d93b"} Jul 29 07:55:30.551099 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:30.551065 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:30.659372 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:30.659277 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00384ee-0fc8-461b-be48-0f1d182838ec-maas-api-service-ca\") pod \"b00384ee-0fc8-461b-be48-0f1d182838ec\" (UID: \"b00384ee-0fc8-461b-be48-0f1d182838ec\") " Jul 29 07:55:30.659709 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:30.659677 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00384ee-0fc8-461b-be48-0f1d182838ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b00384ee-0fc8-461b-be48-0f1d182838ec" (UID: "b00384ee-0fc8-461b-be48-0f1d182838ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:30.760824 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:30.760784 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00384ee-0fc8-461b-be48-0f1d182838ec-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:31.411783 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:31.411754 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" Jul 29 07:55:31.411973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:31.411802 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn" event={"ID":"b00384ee-0fc8-461b-be48-0f1d182838ec","Type":"ContainerDied","Data":"895cd758c7ffd3777dc2c86037b6e2b136e5208b20914101e6bf5f938079e430"} Jul 29 07:55:31.411973 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:31.411853 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="895cd758c7ffd3777dc2c86037b6e2b136e5208b20914101e6bf5f938079e430" Jul 29 07:55:32.427845 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.427815 2568 generic.go:356] "Generic (PLEG): container finished" podID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerID="187c720fea61a63975321a50fd0cadf70ffa53299ded676b81c68df4df576926" exitCode=0 Jul 29 07:55:32.428263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.427888 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" event={"ID":"cb931d56-ed85-4511-8a16-77785f6e64b7","Type":"ContainerDied","Data":"187c720fea61a63975321a50fd0cadf70ffa53299ded676b81c68df4df576926"} Jul 29 07:55:32.428263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.427994 2568 scope.go:117] "RemoveContainer" containerID="9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443" Jul 29 07:55:32.444231 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:55:32.444197 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443\": container with ID starting with 9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443 not found: ID does not exist" containerID="9f6ea4a68fdb3eaaec7c41163cb172d42de99eb224d65ea5b2365ba6e3587443" Jul 29 07:55:32.448244 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.448215 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:55:32.475569 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475526 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb931d56-ed85-4511-8a16-77785f6e64b7-istiod-ca-cert\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475781 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475583 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-credential-socket\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475781 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475709 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-data\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475781 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475769 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-podinfo\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475826 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-token\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475848 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-socket\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475891 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-certs\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475916 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9q4z\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-kube-api-access-v9q4z\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.475969 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-envoy\") pod \"cb931d56-ed85-4511-8a16-77785f6e64b7\" (UID: \"cb931d56-ed85-4511-8a16-77785f6e64b7\") " Jul 29 07:55:32.476263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475889 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:32.476263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.475931 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb931d56-ed85-4511-8a16-77785f6e64b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:32.476263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476178 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:32.476263 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476207 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:32.476465 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476371 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb931d56-ed85-4511-8a16-77785f6e64b7-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.476465 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476389 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.476465 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476404 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.476465 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476415 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.477017 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.476770 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:32.479184 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.479150 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-kube-api-access-v9q4z" (OuterVolumeSpecName: "kube-api-access-v9q4z") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "kube-api-access-v9q4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:32.479573 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.479477 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:55:32.479573 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.479522 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:32.479757 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.479603 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb931d56-ed85-4511-8a16-77785f6e64b7" (UID: "cb931d56-ed85-4511-8a16-77785f6e64b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:32.577531 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.577496 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.577531 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.577527 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.577531 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.577536 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.577776 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.577544 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9q4z\" (UniqueName: \"kubernetes.io/projected/cb931d56-ed85-4511-8a16-77785f6e64b7-kube-api-access-v9q4z\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:32.577776 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:32.577552 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb931d56-ed85-4511-8a16-77785f6e64b7-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:33.436491 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:33.436464 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" Jul 29 07:55:33.436952 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:33.436450 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx" event={"ID":"cb931d56-ed85-4511-8a16-77785f6e64b7","Type":"ContainerDied","Data":"2935b9d3c787f911cc5f3486f6d22f952fd95ec807cdfa0693159a2800328d0a"} Jul 29 07:55:33.436952 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:33.436625 2568 scope.go:117] "RemoveContainer" containerID="187c720fea61a63975321a50fd0cadf70ffa53299ded676b81c68df4df576926" Jul 29 07:55:33.456988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:33.456957 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx"] Jul 29 07:55:33.463716 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:33.463689 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0d455e-openshift-default-77b746bc5d-9q6tx"] Jul 29 07:55:34.738346 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:34.738313 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" path="/var/lib/kubelet/pods/cb931d56-ed85-4511-8a16-77785f6e64b7/volumes" Jul 29 07:55:43.390788 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:43.390744 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:43.609898 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:43.609860 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn"] Jul 29 07:55:43.612359 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:43.612322 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fd8a8c-nnlnn"] Jul 29 07:55:44.738562 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:44.738526 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b00384ee-0fc8-461b-be48-0f1d182838ec" path="/var/lib/kubelet/pods/b00384ee-0fc8-461b-be48-0f1d182838ec/volumes" Jul 29 07:55:45.715609 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:45.715527 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:45.779403 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:45.779354 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:50.252341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:50.252282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:50.305621 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:50.305578 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv"] Jul 29 07:55:50.305844 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:50.305820 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" containerID="cri-o://a3cb5d41565fcf672ed37a538e705dbbc9b3923125032c715e5e07969a723fac" gracePeriod=30 Jul 29 07:55:50.331444 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:50.331410 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:50.823710 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:50.823663 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:50.941828 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:50.941785 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:51.313480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:51.313425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:51.877222 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:51.877180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:54.180616 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:54.180571 2568 patch_prober.go:28] interesting pod/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:55:54.181131 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:54.180639 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:55:54.622129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:54.621959 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:54.739786 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:54.739740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:54.842784 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:54.842751 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:55.051523 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.051480 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:55.350711 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.350679 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:55.565336 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.565289 2568 generic.go:356] "Generic (PLEG): container finished" podID="8435df18-f92c-404d-a80d-6c8e1f586043" containerID="a3cb5d41565fcf672ed37a538e705dbbc9b3923125032c715e5e07969a723fac" exitCode=0 Jul 29 07:55:55.565336 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.565330 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" event={"ID":"8435df18-f92c-404d-a80d-6c8e1f586043","Type":"ContainerDied","Data":"a3cb5d41565fcf672ed37a538e705dbbc9b3923125032c715e5e07969a723fac"} Jul 29 07:55:55.565650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.565397 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" event={"ID":"8435df18-f92c-404d-a80d-6c8e1f586043","Type":"ContainerDied","Data":"182879e181ee9e70d983160dad2e319a910b63448ee6af3837d991dc29e78457"} Jul 29 07:55:55.565650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.565417 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="182879e181ee9e70d983160dad2e319a910b63448ee6af3837d991dc29e78457" Jul 29 07:55:55.565650 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.565433 2568 scope.go:117] "RemoveContainer" containerID="41a3742554c82a0b1b3cc255315113b1c88b9053bcfec492407fb81858a18528" Jul 29 07:55:55.583644 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.583613 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:55:55.688250 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688210 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8435df18-f92c-404d-a80d-6c8e1f586043-istiod-ca-cert\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688269 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-socket\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688286 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-certs\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688414 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-istio-token\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688460 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688457 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gjzn\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-kube-api-access-9gjzn\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688484 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-credential-socket\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688569 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688586 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-envoy\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688614 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-data\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688646 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8435df18-f92c-404d-a80d-6c8e1f586043-istio-podinfo\") pod \"8435df18-f92c-404d-a80d-6c8e1f586043\" (UID: \"8435df18-f92c-404d-a80d-6c8e1f586043\") " Jul 29 07:55:55.688694 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688674 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8435df18-f92c-404d-a80d-6c8e1f586043-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:55.689083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.688718 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:55.689083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.689012 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.689083 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.689034 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8435df18-f92c-404d-a80d-6c8e1f586043-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.689255 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.689088 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.689255 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.689101 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.689553 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.689504 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-data" (OuterVolumeSpecName: "istio-data") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:55.690936 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.690903 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-istio-token" (OuterVolumeSpecName: "istio-token") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:55.691053 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.690929 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:55:55.691122 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.691069 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8435df18-f92c-404d-a80d-6c8e1f586043-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:55:55.691397 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.691373 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-kube-api-access-9gjzn" (OuterVolumeSpecName: "kube-api-access-9gjzn") pod "8435df18-f92c-404d-a80d-6c8e1f586043" (UID: "8435df18-f92c-404d-a80d-6c8e1f586043"). InnerVolumeSpecName "kube-api-access-9gjzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:55.790357 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.790316 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.790357 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.790354 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjzn\" (UniqueName: \"kubernetes.io/projected/8435df18-f92c-404d-a80d-6c8e1f586043-kube-api-access-9gjzn\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.790582 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.790369 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.790582 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.790380 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8435df18-f92c-404d-a80d-6c8e1f586043-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.790582 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.790390 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8435df18-f92c-404d-a80d-6c8e1f586043-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 07:55:55.912149 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:55.912113 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:55:56.573603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:56.573563 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv" Jul 29 07:55:56.611350 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:56.611311 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv"] Jul 29 07:55:56.616019 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:56.615987 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fd8a8c-openshift-default-7f6f6bdb46-r4smv"] Jul 29 07:55:56.738785 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:55:56.738743 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" path="/var/lib/kubelet/pods/8435df18-f92c-404d-a80d-6c8e1f586043/volumes" Jul 29 07:56:19.880899 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:19.880850 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:20.257736 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.257697 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:20.687317 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687271 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595"] Jul 29 07:56:20.687874 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687858 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" Jul 29 07:56:20.687874 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687876 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" Jul 29 07:56:20.688022 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687899 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00384ee-0fc8-461b-be48-0f1d182838ec" containerName="revoke-keys" Jul 29 07:56:20.688022 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687908 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00384ee-0fc8-461b-be48-0f1d182838ec" containerName="revoke-keys" Jul 29 07:56:20.688022 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687932 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" Jul 29 07:56:20.688022 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687941 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" Jul 29 07:56:20.688022 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687956 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" Jul 29 07:56:20.688022 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.687964 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" Jul 29 07:56:20.688341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.688075 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" Jul 29 07:56:20.688341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.688091 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" Jul 29 07:56:20.688341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.688104 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" Jul 29 07:56:20.688341 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.688112 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b00384ee-0fc8-461b-be48-0f1d182838ec" containerName="revoke-keys" Jul 29 07:56:20.693102 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693713 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-istio-token\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b28h\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-kube-api-access-4b28h\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693809 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-credential-socket\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-certs\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693879 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-socket\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.693924 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de31ccd-cb39-474d-914c-236f50281384-istio-podinfo\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.694056 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-data\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.694056 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.693966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-envoy\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.697350 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:20.697319 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="9de31ccd-cb39-474d-914c-236f50281384" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.697350 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:20.697339 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-shared-b-71e038-openshift-default-dockercfg-wmbll\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-71e038-openshift-default-dockercfg-wmbll\"" type="*v1.Secret" Jul 29 07:56:20.697679 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:20.697653 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" Jul 29 07:56:20.740749 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.740714 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595"] Jul 29 07:56:20.794797 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.794753 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.794808 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b28h\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-kube-api-access-4b28h\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.794841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-credential-socket\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.794866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-certs\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.794925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-socket\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.794952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de31ccd-cb39-474d-914c-236f50281384-istio-podinfo\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795021 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-data\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-envoy\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-istio-token\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795300 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-certs\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795300 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-credential-socket\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795485 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795366 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-socket\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.795744 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.795685 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-data\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.797636 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.797604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-envoy\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.797988 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.797963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de31ccd-cb39-474d-914c-236f50281384-istio-podinfo\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.823115 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.823079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-istio-token\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.834480 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.834444 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b28h\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-kube-api-access-4b28h\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:20.845933 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:20.845897 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:21.027172 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:21.027076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:21.795767 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:21.795729 2568 configmap.go:193] Couldn't get configMap openshift-ingress/openshift-gw-ca-root-cert: failed to sync configmap cache: timed out waiting for the condition Jul 29 07:56:21.795962 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:21.795834 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert podName:9de31ccd-cb39-474d-914c-236f50281384 nodeName:}" failed. No retries permitted until 2026-07-29 07:56:22.295811102 +0000 UTC m=+2492.067261879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istiod-ca-cert" (UniqueName: "kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert") pod "e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" (UID: "9de31ccd-cb39-474d-914c-236f50281384") : failed to sync configmap cache: timed out waiting for the condition Jul 29 07:56:21.832267 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:21.832238 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:56:22.305669 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:22.305633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:22.306297 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:22.306276 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert\") pod \"e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:22.447678 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:22.447636 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-71e038-openshift-default-dockercfg-wmbll\"" Jul 29 07:56:22.504239 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:22.504199 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:31.272693 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:31.272635 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595"] Jul 29 07:56:31.275536 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:56:31.275494 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de31ccd_cb39_474d_914c_236f50281384.slice/crio-155639ae5f560a79e0811d014c88772f9677052c9b4980e79ee30f096abbb2f9 WatchSource:0}: Error finding container 155639ae5f560a79e0811d014c88772f9677052c9b4980e79ee30f096abbb2f9: Status 404 returned error can't find the container with id 155639ae5f560a79e0811d014c88772f9677052c9b4980e79ee30f096abbb2f9 Jul 29 07:56:31.277644 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:31.277604 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:56:31.277783 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:31.277694 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:56:31.277783 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:31.277763 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:56:31.772414 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:31.772374 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" event={"ID":"9de31ccd-cb39-474d-914c-236f50281384","Type":"ContainerStarted","Data":"6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939"} Jul 29 07:56:31.772603 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:31.772432 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" event={"ID":"9de31ccd-cb39-474d-914c-236f50281384","Type":"ContainerStarted","Data":"155639ae5f560a79e0811d014c88772f9677052c9b4980e79ee30f096abbb2f9"} Jul 29 07:56:32.505228 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:32.505182 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:32.510640 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:32.510605 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:32.560207 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:32.560154 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podStartSLOduration=12.560135985 podStartE2EDuration="12.560135985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:56:31.802164719 +0000 UTC m=+2501.573615518" watchObservedRunningTime="2026-07-29 07:56:32.560135985 +0000 UTC m=+2502.331586858" Jul 29 07:56:32.777895 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:32.777800 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:32.778932 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:32.778912 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:35.384305 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.384260 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:35.542141 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.542096 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:35.644540 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.644457 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p"] Jul 29 07:56:35.644862 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.644849 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" Jul 29 07:56:35.644918 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.644863 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb931d56-ed85-4511-8a16-77785f6e64b7" containerName="istio-proxy" Jul 29 07:56:35.644966 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.644957 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8435df18-f92c-404d-a80d-6c8e1f586043" containerName="istio-proxy" Jul 29 07:56:35.648208 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.648188 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.652400 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.652380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-71e038-serving-cert\"" Jul 29 07:56:35.672632 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.672597 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p"] Jul 29 07:56:35.707478 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.707439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:35.724556 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.724515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hsf\" (UniqueName: \"kubernetes.io/projected/ddf3fb74-101b-4524-90e7-cc05da325c5c-kube-api-access-94hsf\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.724746 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.724563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf3fb74-101b-4524-90e7-cc05da325c5c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.724746 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.724690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.825672 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.825634 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hsf\" (UniqueName: \"kubernetes.io/projected/ddf3fb74-101b-4524-90e7-cc05da325c5c-kube-api-access-94hsf\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.825672 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.825677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf3fb74-101b-4524-90e7-cc05da325c5c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.825947 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.825898 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.826185 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:35.826152 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-serving-cert: secret "maas-api-e2e-shared-b-71e038-serving-cert" not found Jul 29 07:56:35.826312 ip-10-0-136-73 kubenswrapper[2568]: E0729 07:56:35.826261 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls podName:ddf3fb74-101b-4524-90e7-cc05da325c5c nodeName:}" failed. No retries permitted until 2026-07-29 07:56:36.326240702 +0000 UTC m=+2506.097691482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls") pod "maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" (UID: "ddf3fb74-101b-4524-90e7-cc05da325c5c") : secret "maas-api-e2e-shared-b-71e038-serving-cert" not found Jul 29 07:56:35.826445 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.826425 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf3fb74-101b-4524-90e7-cc05da325c5c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:35.841887 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:35.841847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hsf\" (UniqueName: \"kubernetes.io/projected/ddf3fb74-101b-4524-90e7-cc05da325c5c-kube-api-access-94hsf\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:36.006311 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.006278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:36.332432 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.332327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:36.335031 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.335001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-71e038-6ddb457886-btw4p\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:36.558745 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.558702 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:36.738951 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.738891 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:36.741977 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.741948 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p"] Jul 29 07:56:36.745482 ip-10-0-136-73 kubenswrapper[2568]: W0729 07:56:36.745455 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf3fb74_101b_4524_90e7_cc05da325c5c.slice/crio-75f78ec62605870ca3869b87bd4ef0e566632816b66dae92744f1a4cddbfb4a1 WatchSource:0}: Error finding container 75f78ec62605870ca3869b87bd4ef0e566632816b66dae92744f1a4cddbfb4a1: Status 404 returned error can't find the container with id 75f78ec62605870ca3869b87bd4ef0e566632816b66dae92744f1a4cddbfb4a1 Jul 29 07:56:36.747185 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.747169 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:56:36.799319 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.799283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" event={"ID":"ddf3fb74-101b-4524-90e7-cc05da325c5c","Type":"ContainerStarted","Data":"75f78ec62605870ca3869b87bd4ef0e566632816b66dae92744f1a4cddbfb4a1"} Jul 29 07:56:36.898270 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:36.898177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:56:37.807199 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:37.807161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" event={"ID":"ddf3fb74-101b-4524-90e7-cc05da325c5c","Type":"ContainerStarted","Data":"d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c"} Jul 29 07:56:37.807597 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:37.807235 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:37.831140 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:37.831073 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" podStartSLOduration=2.362156408 podStartE2EDuration="2.831031614s" totalImagesPullingTime="468.875206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:56:37.828967702 +0000 UTC m=+2507.600418497" watchObservedRunningTime="2026-07-29 07:56:37.831031614 +0000 UTC m=+2507.602482414" Jul 29 07:56:43.826449 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:43.826353 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 07:56:47.779006 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.778972 2568 patch_prober.go:28] interesting pod/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 29 07:56:47.779483 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.779029 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 29 07:56:47.882265 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.882172 2568 generic.go:356] "Generic (PLEG): container finished" podID="9de31ccd-cb39-474d-914c-236f50281384" containerID="6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939" exitCode=0 Jul 29 07:56:47.882265 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.882248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" event={"ID":"9de31ccd-cb39-474d-914c-236f50281384","Type":"ContainerDied","Data":"6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939"} Jul 29 07:56:47.882712 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.882698 2568 scope.go:117] "RemoveContainer" containerID="6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939" Jul 29 07:56:47.883852 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.883824 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:56:47.883934 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.883919 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:56:47.883977 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:47.883964 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 07:56:48.890399 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:48.890365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" event={"ID":"9de31ccd-cb39-474d-914c-236f50281384","Type":"ContainerStarted","Data":"7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b"} Jul 29 07:56:49.504943 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:49.504908 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:49.505292 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:49.505264 2568 patch_prober.go:28] interesting pod/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 29 07:56:49.505356 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:49.505338 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 29 07:56:50.505349 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:50.505311 2568 patch_prober.go:28] interesting pod/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 29 07:56:50.505778 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:50.505370 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 29 07:56:51.504820 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:51.504779 2568 patch_prober.go:28] interesting pod/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 29 07:56:51.505015 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:51.504841 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 29 07:56:52.505279 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:52.505230 2568 patch_prober.go:28] interesting pod/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 29 07:56:52.505702 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:52.505297 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 29 07:56:53.509671 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:53.509640 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:53.510117 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:53.509926 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:53.510663 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:53.510646 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 07:56:59.510518 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:56:59.510477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:57:00.515360 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:57:00.515328 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:57:06.539412 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:57:06.539371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:57:13.505476 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:57:13.505434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:57:16.914129 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:57:16.914083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:57:54.498415 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:57:54.498374 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:59:19.654384 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:59:19.654342 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:59:19.762727 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:59:19.762692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 07:59:50.895837 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:59:50.895754 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:59:50.906103 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:59:50.906078 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 07:59:51.180036 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:59:51.179997 2568 scope.go:117] "RemoveContainer" containerID="aa79cfb46799dd90b79db746b2802cc049020022f83123fcd260e60129f0bd85" Jul 29 07:59:51.189774 ip-10-0-136-73 kubenswrapper[2568]: I0729 07:59:51.189749 2568 scope.go:117] "RemoveContainer" containerID="7a4b19da5a13020861ddf77c3eb076364d94411f62b8a046abf23958bdb9d602" Jul 29 08:00:00.148132 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.148090 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2"] Jul 29 08:00:00.152079 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.152034 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" Jul 29 08:00:00.152355 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.152332 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx"] Jul 29 08:00:00.154933 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.154906 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 08:00:00.155818 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.155800 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj"] Jul 29 08:00:00.155947 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.155931 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" Jul 29 08:00:00.159425 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.159393 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" Jul 29 08:00:00.164360 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.164334 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" Jul 29 08:00:00.172163 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.172127 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" Jul 29 08:00:00.180513 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.180458 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj"] Jul 29 08:00:00.182951 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.182907 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx"] Jul 29 08:00:00.206713 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.206511 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" Jul 29 08:00:00.207385 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.207358 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2"] Jul 29 08:00:00.360708 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.360636 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2"] Jul 29 08:00:00.362313 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:00:00.362280 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b35d97_ea13_4fb2_b5e4_c3737e8bce08.slice/crio-3d99b53ddbe4f5ed72914a4473b4147df5eaa3ea1bd21abd556c78a91355e3b4 WatchSource:0}: Error finding container 3d99b53ddbe4f5ed72914a4473b4147df5eaa3ea1bd21abd556c78a91355e3b4: Status 404 returned error can't find the container with id 3d99b53ddbe4f5ed72914a4473b4147df5eaa3ea1bd21abd556c78a91355e3b4 Jul 29 08:00:00.603462 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.603436 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj"] Jul 29 08:00:00.605853 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:00:00.605802 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0325e2e1_f143_44fd_9c8b_f69bafac1024.slice/crio-b7ffe0351e74183b78970fee95048c7a2070adce35f7b0db051ce2bbbd49b425 WatchSource:0}: Error finding container b7ffe0351e74183b78970fee95048c7a2070adce35f7b0db051ce2bbbd49b425: Status 404 returned error can't find the container with id b7ffe0351e74183b78970fee95048c7a2070adce35f7b0db051ce2bbbd49b425 Jul 29 08:00:00.607409 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.607385 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx"] Jul 29 08:00:00.610122 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:00:00.610092 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73ef970_2e12_446a_873d_0840a315813b.slice/crio-f015e30a5d896f40bf087adce525ce2e8eac9c82714fb60937e2aa8055c731f1 WatchSource:0}: Error finding container f015e30a5d896f40bf087adce525ce2e8eac9c82714fb60937e2aa8055c731f1: Status 404 returned error can't find the container with id f015e30a5d896f40bf087adce525ce2e8eac9c82714fb60937e2aa8055c731f1 Jul 29 08:00:00.981209 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.981176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" event={"ID":"0325e2e1-f143-44fd-9c8b-f69bafac1024","Type":"ContainerStarted","Data":"6a64ea6d9a44a7ca58bfdb88c6570d32cc51bd8a1704da870beb737a92f5164a"} Jul 29 08:00:00.981443 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.981234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" event={"ID":"0325e2e1-f143-44fd-9c8b-f69bafac1024","Type":"ContainerStarted","Data":"b7ffe0351e74183b78970fee95048c7a2070adce35f7b0db051ce2bbbd49b425"} Jul 29 08:00:00.982456 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.982425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" event={"ID":"d73ef970-2e12-446a-873d-0840a315813b","Type":"ContainerStarted","Data":"f015e30a5d896f40bf087adce525ce2e8eac9c82714fb60937e2aa8055c731f1"} Jul 29 08:00:00.983908 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.983880 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" event={"ID":"42b35d97-ea13-4fb2-b5e4-c3737e8bce08","Type":"ContainerStarted","Data":"e2fb140dd3a06eae1ada07f47f23199f73302841ba503b52b05be5cdceca90e9"} Jul 29 08:00:00.984032 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:00.983935 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" event={"ID":"42b35d97-ea13-4fb2-b5e4-c3737e8bce08","Type":"ContainerStarted","Data":"3d99b53ddbe4f5ed72914a4473b4147df5eaa3ea1bd21abd556c78a91355e3b4"} Jul 29 08:00:01.014220 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.014073 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" podStartSLOduration=1.014020883 podStartE2EDuration="1.014020883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:00:00.997719324 +0000 UTC m=+2710.769170123" watchObservedRunningTime="2026-07-29 08:00:01.014020883 +0000 UTC m=+2710.785471684" Jul 29 08:00:01.015032 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.014998 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" podStartSLOduration=1.014989624 podStartE2EDuration="1.014989624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:00:01.014016368 +0000 UTC m=+2710.785467168" watchObservedRunningTime="2026-07-29 08:00:01.014989624 +0000 UTC m=+2710.786440425" Jul 29 08:00:01.991257 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.991220 2568 generic.go:356] "Generic (PLEG): container finished" podID="0325e2e1-f143-44fd-9c8b-f69bafac1024" containerID="6a64ea6d9a44a7ca58bfdb88c6570d32cc51bd8a1704da870beb737a92f5164a" exitCode=0 Jul 29 08:00:01.991735 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.991307 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" event={"ID":"0325e2e1-f143-44fd-9c8b-f69bafac1024","Type":"ContainerDied","Data":"6a64ea6d9a44a7ca58bfdb88c6570d32cc51bd8a1704da870beb737a92f5164a"} Jul 29 08:00:01.992949 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.992915 2568 generic.go:356] "Generic (PLEG): container finished" podID="d73ef970-2e12-446a-873d-0840a315813b" containerID="d6383957fbb184fa3de7a36474997bece69ecadc3edbb3278d35f22fdfcea95b" exitCode=0 Jul 29 08:00:01.993115 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.992989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" event={"ID":"d73ef970-2e12-446a-873d-0840a315813b","Type":"ContainerDied","Data":"d6383957fbb184fa3de7a36474997bece69ecadc3edbb3278d35f22fdfcea95b"} Jul 29 08:00:01.994607 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.994585 2568 generic.go:356] "Generic (PLEG): container finished" podID="42b35d97-ea13-4fb2-b5e4-c3737e8bce08" containerID="e2fb140dd3a06eae1ada07f47f23199f73302841ba503b52b05be5cdceca90e9" exitCode=0 Jul 29 08:00:01.994720 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:01.994671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" event={"ID":"42b35d97-ea13-4fb2-b5e4-c3737e8bce08","Type":"ContainerDied","Data":"e2fb140dd3a06eae1ada07f47f23199f73302841ba503b52b05be5cdceca90e9"} Jul 29 08:00:03.174104 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:03.174074 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" Jul 29 08:00:03.223410 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:03.223374 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" Jul 29 08:00:03.227909 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:03.227880 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" Jul 29 08:00:04.009526 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.009490 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" Jul 29 08:00:04.009750 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.009489 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj" event={"ID":"0325e2e1-f143-44fd-9c8b-f69bafac1024","Type":"ContainerDied","Data":"b7ffe0351e74183b78970fee95048c7a2070adce35f7b0db051ce2bbbd49b425"} Jul 29 08:00:04.009750 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.009618 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ffe0351e74183b78970fee95048c7a2070adce35f7b0db051ce2bbbd49b425" Jul 29 08:00:04.011182 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.011157 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" Jul 29 08:00:04.011357 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.011165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755200-jfwpx" event={"ID":"d73ef970-2e12-446a-873d-0840a315813b","Type":"ContainerDied","Data":"f015e30a5d896f40bf087adce525ce2e8eac9c82714fb60937e2aa8055c731f1"} Jul 29 08:00:04.011357 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.011281 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f015e30a5d896f40bf087adce525ce2e8eac9c82714fb60937e2aa8055c731f1" Jul 29 08:00:04.017062 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.013476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" event={"ID":"42b35d97-ea13-4fb2-b5e4-c3737e8bce08","Type":"ContainerDied","Data":"3d99b53ddbe4f5ed72914a4473b4147df5eaa3ea1bd21abd556c78a91355e3b4"} Jul 29 08:00:04.017062 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.013559 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d99b53ddbe4f5ed72914a4473b4147df5eaa3ea1bd21abd556c78a91355e3b4" Jul 29 08:00:04.017062 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:04.013627 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2" Jul 29 08:00:51.216592 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:00:51.216547 2568 scope.go:117] "RemoveContainer" containerID="a3cb5d41565fcf672ed37a538e705dbbc9b3923125032c715e5e07969a723fac" Jul 29 08:01:43.501316 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:01:43.501276 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:01:51.242230 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:01:51.242189 2568 scope.go:117] "RemoveContainer" containerID="8dd5050c604cc20c3e6133a11ac523fbb961c4a09793b5c7e4de7361a308d93b" Jul 29 08:01:51.252090 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:01:51.252035 2568 scope.go:117] "RemoveContainer" containerID="1f6a698a00fa49503fc4be437af02c1f24f31ce9f1ccf88860b7be352998e376" Jul 29 08:02:34.136157 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:02:34.136117 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:02:35.042220 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:02:35.042181 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:02:45.620626 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:02:45.620586 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:03:01.328104 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:03:01.328064 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:03:01.388600 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:03:01.388562 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:03:11.729398 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:03:11.729360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:03:26.774615 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:03:26.774572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:04:30.916729 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:30.916667 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:04:32.036352 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:32.036302 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:04:50.147611 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:50.147548 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:04:50.211791 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:50.211736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:04:50.427008 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:50.426961 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:04:50.938289 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:50.938257 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 08:04:50.947987 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:04:50.947959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 08:05:24.743864 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:24.743827 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:05:40.236237 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236197 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9z7zz"] Jul 29 08:05:40.236684 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236646 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b35d97-ea13-4fb2-b5e4-c3737e8bce08" containerName="cleanup" Jul 29 08:05:40.236684 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236660 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b35d97-ea13-4fb2-b5e4-c3737e8bce08" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236691 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0325e2e1-f143-44fd-9c8b-f69bafac1024" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236697 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0325e2e1-f143-44fd-9c8b-f69bafac1024" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236711 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73ef970-2e12-446a-873d-0840a315813b" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236717 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73ef970-2e12-446a-873d-0840a315813b" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236784 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0325e2e1-f143-44fd-9c8b-f69bafac1024" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236791 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b35d97-ea13-4fb2-b5e4-c3737e8bce08" containerName="cleanup" Jul 29 08:05:40.236812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.236798 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73ef970-2e12-446a-873d-0840a315813b" containerName="cleanup" Jul 29 08:05:40.240327 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.240305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:40.254147 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.254110 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9z7zz"] Jul 29 08:05:40.383393 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.383352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkt2\" (UniqueName: \"kubernetes.io/projected/6f41ec29-0d47-4473-a41e-7913201fc91c-kube-api-access-7dkt2\") pod \"kuadrant-operator-catalog-9z7zz\" (UID: \"6f41ec29-0d47-4473-a41e-7913201fc91c\") " pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:40.484421 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.484366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkt2\" (UniqueName: \"kubernetes.io/projected/6f41ec29-0d47-4473-a41e-7913201fc91c-kube-api-access-7dkt2\") pod \"kuadrant-operator-catalog-9z7zz\" (UID: \"6f41ec29-0d47-4473-a41e-7913201fc91c\") " pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:40.495739 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.495662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkt2\" (UniqueName: \"kubernetes.io/projected/6f41ec29-0d47-4473-a41e-7913201fc91c-kube-api-access-7dkt2\") pod \"kuadrant-operator-catalog-9z7zz\" (UID: \"6f41ec29-0d47-4473-a41e-7913201fc91c\") " pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:40.552240 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.552192 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:40.707028 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.706990 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9z7zz"] Jul 29 08:05:40.708623 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:05:40.708594 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f41ec29_0d47_4473_a41e_7913201fc91c.slice/crio-38160e96f345e245321605d723283546182f26ff66a9bd0b3e12430a702dc14d WatchSource:0}: Error finding container 38160e96f345e245321605d723283546182f26ff66a9bd0b3e12430a702dc14d: Status 404 returned error can't find the container with id 38160e96f345e245321605d723283546182f26ff66a9bd0b3e12430a702dc14d Jul 29 08:05:40.710027 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.710006 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:05:40.927994 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:40.927959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" event={"ID":"6f41ec29-0d47-4473-a41e-7913201fc91c","Type":"ContainerStarted","Data":"38160e96f345e245321605d723283546182f26ff66a9bd0b3e12430a702dc14d"} Jul 29 08:05:42.941330 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:42.941273 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" event={"ID":"6f41ec29-0d47-4473-a41e-7913201fc91c","Type":"ContainerStarted","Data":"288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3"} Jul 29 08:05:42.961180 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:42.961122 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" podStartSLOduration=1.723646926 podStartE2EDuration="2.961103486s" totalImagesPullingTime="1.23745656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:05:42.959698223 +0000 UTC m=+3052.731149024" watchObservedRunningTime="2026-07-29 08:05:42.961103486 +0000 UTC m=+3052.732554284" Jul 29 08:05:50.553368 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:50.553323 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:50.553802 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:50.553389 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:50.576425 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:50.576393 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:51.016616 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:51.016578 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:51.160157 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:51.160116 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9z7zz"] Jul 29 08:05:52.997781 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:52.997743 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" podUID="6f41ec29-0d47-4473-a41e-7913201fc91c" containerName="registry-server" containerID="cri-o://288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3" gracePeriod=2 Jul 29 08:05:53.266964 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:53.266924 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:53.405826 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:53.405782 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dkt2\" (UniqueName: \"kubernetes.io/projected/6f41ec29-0d47-4473-a41e-7913201fc91c-kube-api-access-7dkt2\") pod \"6f41ec29-0d47-4473-a41e-7913201fc91c\" (UID: \"6f41ec29-0d47-4473-a41e-7913201fc91c\") " Jul 29 08:05:53.408150 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:53.408118 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f41ec29-0d47-4473-a41e-7913201fc91c-kube-api-access-7dkt2" (OuterVolumeSpecName: "kube-api-access-7dkt2") pod "6f41ec29-0d47-4473-a41e-7913201fc91c" (UID: "6f41ec29-0d47-4473-a41e-7913201fc91c"). InnerVolumeSpecName "kube-api-access-7dkt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:05:53.507578 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:53.507484 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dkt2\" (UniqueName: \"kubernetes.io/projected/6f41ec29-0d47-4473-a41e-7913201fc91c-kube-api-access-7dkt2\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:05:54.004016 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.003982 2568 generic.go:356] "Generic (PLEG): container finished" podID="6f41ec29-0d47-4473-a41e-7913201fc91c" containerID="288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3" exitCode=0 Jul 29 08:05:54.004516 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.004027 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" Jul 29 08:05:54.004516 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.004066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" event={"ID":"6f41ec29-0d47-4473-a41e-7913201fc91c","Type":"ContainerDied","Data":"288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3"} Jul 29 08:05:54.004516 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.004123 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9z7zz" event={"ID":"6f41ec29-0d47-4473-a41e-7913201fc91c","Type":"ContainerDied","Data":"38160e96f345e245321605d723283546182f26ff66a9bd0b3e12430a702dc14d"} Jul 29 08:05:54.004516 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.004145 2568 scope.go:117] "RemoveContainer" containerID="288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3" Jul 29 08:05:54.013996 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.013970 2568 scope.go:117] "RemoveContainer" containerID="288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3" Jul 29 08:05:54.014447 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:05:54.014427 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3\": container with ID starting with 288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3 not found: ID does not exist" containerID="288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3" Jul 29 08:05:54.014528 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.014457 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3"} err="failed to get container status \"288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3\": rpc error: code = NotFound desc = could not find container \"288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3\": container with ID starting with 288a86301c2a31883f9d98b917237f5b2353e3dd65241368b870663950c0a9b3 not found: ID does not exist" Jul 29 08:05:54.033643 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.033604 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9z7zz"] Jul 29 08:05:54.044178 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.044141 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9z7zz"] Jul 29 08:05:54.742576 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:05:54.742540 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f41ec29-0d47-4473-a41e-7913201fc91c" path="/var/lib/kubelet/pods/6f41ec29-0d47-4473-a41e-7913201fc91c/volumes" Jul 29 08:06:35.766210 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:35.766174 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:06:36.617117 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:36.617078 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:06:37.726229 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.726199 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7802-140618630354096"] Jul 29 08:06:37.726657 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.726645 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f41ec29-0d47-4473-a41e-7913201fc91c" containerName="registry-server" Jul 29 08:06:37.726708 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.726658 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f41ec29-0d47-4473-a41e-7913201fc91c" containerName="registry-server" Jul 29 08:06:37.726743 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.726719 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f41ec29-0d47-4473-a41e-7913201fc91c" containerName="registry-server" Jul 29 08:06:37.729698 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.729682 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:37.732763 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.732746 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m4vmz\"" Jul 29 08:06:37.737159 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.737135 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618630354096"] Jul 29 08:06:37.807216 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.807190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jx2\" (UniqueName: \"kubernetes.io/projected/523c7d49-fd88-41e4-ab02-3e92db2a3016-kube-api-access-q6jx2\") pod \"test-curl-7802-140618630354096\" (UID: \"523c7d49-fd88-41e4-ab02-3e92db2a3016\") " pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:37.908418 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.908396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jx2\" (UniqueName: \"kubernetes.io/projected/523c7d49-fd88-41e4-ab02-3e92db2a3016-kube-api-access-q6jx2\") pod \"test-curl-7802-140618630354096\" (UID: \"523c7d49-fd88-41e4-ab02-3e92db2a3016\") " pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:37.917715 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:37.917686 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jx2\" (UniqueName: \"kubernetes.io/projected/523c7d49-fd88-41e4-ab02-3e92db2a3016-kube-api-access-q6jx2\") pod \"test-curl-7802-140618630354096\" (UID: \"523c7d49-fd88-41e4-ab02-3e92db2a3016\") " pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:38.042355 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:38.042283 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:38.182076 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:38.182054 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618630354096"] Jul 29 08:06:38.184025 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:06:38.183998 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod523c7d49_fd88_41e4_ab02_3e92db2a3016.slice/crio-7c22ab606f974086f8c6c81926b3c42b0b24a7a68ba668df81329491d23b8bc2 WatchSource:0}: Error finding container 7c22ab606f974086f8c6c81926b3c42b0b24a7a68ba668df81329491d23b8bc2: Status 404 returned error can't find the container with id 7c22ab606f974086f8c6c81926b3c42b0b24a7a68ba668df81329491d23b8bc2 Jul 29 08:06:38.249235 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:38.249196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618630354096" event={"ID":"523c7d49-fd88-41e4-ab02-3e92db2a3016","Type":"ContainerStarted","Data":"7c22ab606f974086f8c6c81926b3c42b0b24a7a68ba668df81329491d23b8bc2"} Jul 29 08:06:39.256646 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:39.256608 2568 generic.go:356] "Generic (PLEG): container finished" podID="523c7d49-fd88-41e4-ab02-3e92db2a3016" containerID="6995c716d0eddeadcf8381baaa583987fbe1ff05a73c1f8aa5f3fbe4d3e6b880" exitCode=0 Jul 29 08:06:39.257013 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:39.256658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618630354096" event={"ID":"523c7d49-fd88-41e4-ab02-3e92db2a3016","Type":"ContainerDied","Data":"6995c716d0eddeadcf8381baaa583987fbe1ff05a73c1f8aa5f3fbe4d3e6b880"} Jul 29 08:06:40.402432 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.402408 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:40.425762 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.425736 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7802-140618630354096_523c7d49-fd88-41e4-ab02-3e92db2a3016/test-curl-7802-140618630354096/0.log" Jul 29 08:06:40.454108 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.454087 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7802-140618630354096"] Jul 29 08:06:40.457910 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.457889 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7802-140618630354096"] Jul 29 08:06:40.531084 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.530989 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6jx2\" (UniqueName: \"kubernetes.io/projected/523c7d49-fd88-41e4-ab02-3e92db2a3016-kube-api-access-q6jx2\") pod \"523c7d49-fd88-41e4-ab02-3e92db2a3016\" (UID: \"523c7d49-fd88-41e4-ab02-3e92db2a3016\") " Jul 29 08:06:40.533496 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.533471 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523c7d49-fd88-41e4-ab02-3e92db2a3016-kube-api-access-q6jx2" (OuterVolumeSpecName: "kube-api-access-q6jx2") pod "523c7d49-fd88-41e4-ab02-3e92db2a3016" (UID: "523c7d49-fd88-41e4-ab02-3e92db2a3016"). InnerVolumeSpecName "kube-api-access-q6jx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:06:40.594870 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.594844 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7802-140618630356016"] Jul 29 08:06:40.595315 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.595300 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523c7d49-fd88-41e4-ab02-3e92db2a3016" containerName="test-curl-7802-140618630354096" Jul 29 08:06:40.595366 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.595318 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="523c7d49-fd88-41e4-ab02-3e92db2a3016" containerName="test-curl-7802-140618630354096" Jul 29 08:06:40.595402 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.595393 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="523c7d49-fd88-41e4-ab02-3e92db2a3016" containerName="test-curl-7802-140618630354096" Jul 29 08:06:40.598666 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.598648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:40.605297 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.605275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618630356016"] Jul 29 08:06:40.632285 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.632264 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jx2\" (UniqueName: \"kubernetes.io/projected/523c7d49-fd88-41e4-ab02-3e92db2a3016-kube-api-access-q6jx2\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:06:40.733205 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.733169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smcj\" (UniqueName: \"kubernetes.io/projected/d680c5be-d29b-4d61-a560-54549232f33a-kube-api-access-6smcj\") pod \"test-curl-7802-140618630356016\" (UID: \"d680c5be-d29b-4d61-a560-54549232f33a\") " pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:40.739526 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.739496 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523c7d49-fd88-41e4-ab02-3e92db2a3016" path="/var/lib/kubelet/pods/523c7d49-fd88-41e4-ab02-3e92db2a3016/volumes" Jul 29 08:06:40.834621 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.834551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smcj\" (UniqueName: \"kubernetes.io/projected/d680c5be-d29b-4d61-a560-54549232f33a-kube-api-access-6smcj\") pod \"test-curl-7802-140618630356016\" (UID: \"d680c5be-d29b-4d61-a560-54549232f33a\") " pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:40.846406 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.846381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smcj\" (UniqueName: \"kubernetes.io/projected/d680c5be-d29b-4d61-a560-54549232f33a-kube-api-access-6smcj\") pod \"test-curl-7802-140618630356016\" (UID: \"d680c5be-d29b-4d61-a560-54549232f33a\") " pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:40.909174 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:40.909140 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:41.053080 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:41.053035 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618630356016"] Jul 29 08:06:41.054366 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:06:41.054342 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd680c5be_d29b_4d61_a560_54549232f33a.slice/crio-4498624ac1943d66407ff4f5cd85ec063a8034497cca5f3bad9fac25da734874 WatchSource:0}: Error finding container 4498624ac1943d66407ff4f5cd85ec063a8034497cca5f3bad9fac25da734874: Status 404 returned error can't find the container with id 4498624ac1943d66407ff4f5cd85ec063a8034497cca5f3bad9fac25da734874 Jul 29 08:06:41.270485 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:41.270454 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630354096" Jul 29 08:06:41.270734 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:41.270460 2568 scope.go:117] "RemoveContainer" containerID="6995c716d0eddeadcf8381baaa583987fbe1ff05a73c1f8aa5f3fbe4d3e6b880" Jul 29 08:06:41.271978 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:41.271893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618630356016" event={"ID":"d680c5be-d29b-4d61-a560-54549232f33a","Type":"ContainerStarted","Data":"4498624ac1943d66407ff4f5cd85ec063a8034497cca5f3bad9fac25da734874"} Jul 29 08:06:42.278867 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:42.278827 2568 generic.go:356] "Generic (PLEG): container finished" podID="d680c5be-d29b-4d61-a560-54549232f33a" containerID="07b8805ace1e8117cfcc79578b86b6f5921da0d170c6fe163f1134db50ff8070" exitCode=0 Jul 29 08:06:42.279278 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:42.278930 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618630356016" event={"ID":"d680c5be-d29b-4d61-a560-54549232f33a","Type":"ContainerDied","Data":"07b8805ace1e8117cfcc79578b86b6f5921da0d170c6fe163f1134db50ff8070"} Jul 29 08:06:43.429533 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.429504 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:43.452724 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.452698 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7802-140618630356016_d680c5be-d29b-4d61-a560-54549232f33a/test-curl-7802-140618630356016/0.log" Jul 29 08:06:43.484981 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.484949 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7802-140618630356016"] Jul 29 08:06:43.489629 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.489605 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7802-140618630356016"] Jul 29 08:06:43.558993 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.558929 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6smcj\" (UniqueName: \"kubernetes.io/projected/d680c5be-d29b-4d61-a560-54549232f33a-kube-api-access-6smcj\") pod \"d680c5be-d29b-4d61-a560-54549232f33a\" (UID: \"d680c5be-d29b-4d61-a560-54549232f33a\") " Jul 29 08:06:43.561140 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.561116 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d680c5be-d29b-4d61-a560-54549232f33a-kube-api-access-6smcj" (OuterVolumeSpecName: "kube-api-access-6smcj") pod "d680c5be-d29b-4d61-a560-54549232f33a" (UID: "d680c5be-d29b-4d61-a560-54549232f33a"). InnerVolumeSpecName "kube-api-access-6smcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:06:43.659809 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.659779 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smcj\" (UniqueName: \"kubernetes.io/projected/d680c5be-d29b-4d61-a560-54549232f33a-kube-api-access-6smcj\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:06:43.682415 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.682386 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:43.682790 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.682778 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d680c5be-d29b-4d61-a560-54549232f33a" containerName="test-curl-7802-140618630356016" Jul 29 08:06:43.682843 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.682792 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d680c5be-d29b-4d61-a560-54549232f33a" containerName="test-curl-7802-140618630356016" Jul 29 08:06:43.682876 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.682869 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d680c5be-d29b-4d61-a560-54549232f33a" containerName="test-curl-7802-140618630356016" Jul 29 08:06:43.686035 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.686019 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:43.693139 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.692927 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:43.760700 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.760671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz76h\" (UniqueName: \"kubernetes.io/projected/23875595-4c57-4b45-873f-37fb0ad7e6ff-kube-api-access-cz76h\") pod \"test-curl-7802-140618635221296\" (UID: \"23875595-4c57-4b45-873f-37fb0ad7e6ff\") " pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:43.861931 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.861848 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz76h\" (UniqueName: \"kubernetes.io/projected/23875595-4c57-4b45-873f-37fb0ad7e6ff-kube-api-access-cz76h\") pod \"test-curl-7802-140618635221296\" (UID: \"23875595-4c57-4b45-873f-37fb0ad7e6ff\") " pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:43.872703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.872664 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz76h\" (UniqueName: \"kubernetes.io/projected/23875595-4c57-4b45-873f-37fb0ad7e6ff-kube-api-access-cz76h\") pod \"test-curl-7802-140618635221296\" (UID: \"23875595-4c57-4b45-873f-37fb0ad7e6ff\") " pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:43.996021 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:43.995996 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:44.135305 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:44.135270 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:44.135440 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:06:44.135346 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23875595_4c57_4b45_873f_37fb0ad7e6ff.slice/crio-7a2e48ef11a7a37237945e0f0a16889f219c27796e891b4db44398d8c01150f8 WatchSource:0}: Error finding container 7a2e48ef11a7a37237945e0f0a16889f219c27796e891b4db44398d8c01150f8: Status 404 returned error can't find the container with id 7a2e48ef11a7a37237945e0f0a16889f219c27796e891b4db44398d8c01150f8 Jul 29 08:06:44.291936 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:44.291906 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:44.292117 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:44.291949 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4498624ac1943d66407ff4f5cd85ec063a8034497cca5f3bad9fac25da734874" Jul 29 08:06:44.293220 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:44.293187 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618635221296" event={"ID":"23875595-4c57-4b45-873f-37fb0ad7e6ff","Type":"ContainerStarted","Data":"7a2e48ef11a7a37237945e0f0a16889f219c27796e891b4db44398d8c01150f8"} Jul 29 08:06:44.303326 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:06:44.303300 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7802-140618630356016\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="d680c5be-d29b-4d61-a560-54549232f33a" pod="opendatahub/test-curl-7802-140618630356016" Jul 29 08:06:44.739257 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:44.739186 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d680c5be-d29b-4d61-a560-54549232f33a" path="/var/lib/kubelet/pods/d680c5be-d29b-4d61-a560-54549232f33a/volumes" Jul 29 08:06:45.300105 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:45.300075 2568 generic.go:356] "Generic (PLEG): container finished" podID="23875595-4c57-4b45-873f-37fb0ad7e6ff" containerID="12cf1504f701b85bf913502467fb66b57dbf142ae5ba45d8245430bdaa7d4362" exitCode=0 Jul 29 08:06:45.300285 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:45.300176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618635221296" event={"ID":"23875595-4c57-4b45-873f-37fb0ad7e6ff","Type":"ContainerDied","Data":"12cf1504f701b85bf913502467fb66b57dbf142ae5ba45d8245430bdaa7d4362"} Jul 29 08:06:46.447750 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.447725 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:46.502794 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.502763 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7802-140618635221296_23875595-4c57-4b45-873f-37fb0ad7e6ff/test-curl-7802-140618635221296/0.log" Jul 29 08:06:46.542107 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.542077 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:46.546132 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.546111 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:46.584696 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.584639 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz76h\" (UniqueName: \"kubernetes.io/projected/23875595-4c57-4b45-873f-37fb0ad7e6ff-kube-api-access-cz76h\") pod \"23875595-4c57-4b45-873f-37fb0ad7e6ff\" (UID: \"23875595-4c57-4b45-873f-37fb0ad7e6ff\") " Jul 29 08:06:46.586841 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.586817 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23875595-4c57-4b45-873f-37fb0ad7e6ff-kube-api-access-cz76h" (OuterVolumeSpecName: "kube-api-access-cz76h") pod "23875595-4c57-4b45-873f-37fb0ad7e6ff" (UID: "23875595-4c57-4b45-873f-37fb0ad7e6ff"). InnerVolumeSpecName "kube-api-access-cz76h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:06:46.685668 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.685643 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz76h\" (UniqueName: \"kubernetes.io/projected/23875595-4c57-4b45-873f-37fb0ad7e6ff-kube-api-access-cz76h\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:06:46.689452 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.689430 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:46.689844 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.689831 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23875595-4c57-4b45-873f-37fb0ad7e6ff" containerName="test-curl-7802-140618635221296" Jul 29 08:06:46.689885 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.689846 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="23875595-4c57-4b45-873f-37fb0ad7e6ff" containerName="test-curl-7802-140618635221296" Jul 29 08:06:46.689941 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.689931 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="23875595-4c57-4b45-873f-37fb0ad7e6ff" containerName="test-curl-7802-140618635221296" Jul 29 08:06:46.692961 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.692941 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:46.703592 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.703569 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:46.738298 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.738268 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23875595-4c57-4b45-873f-37fb0ad7e6ff" path="/var/lib/kubelet/pods/23875595-4c57-4b45-873f-37fb0ad7e6ff/volumes" Jul 29 08:06:46.786658 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.786629 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwdh\" (UniqueName: \"kubernetes.io/projected/88177313-1d69-4b29-8351-cc7ec39a2e1b-kube-api-access-lwwdh\") pod \"test-curl-7802-140618635221296\" (UID: \"88177313-1d69-4b29-8351-cc7ec39a2e1b\") " pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:46.888271 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.888164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwdh\" (UniqueName: \"kubernetes.io/projected/88177313-1d69-4b29-8351-cc7ec39a2e1b-kube-api-access-lwwdh\") pod \"test-curl-7802-140618635221296\" (UID: \"88177313-1d69-4b29-8351-cc7ec39a2e1b\") " pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:46.898888 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:46.898860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwdh\" (UniqueName: \"kubernetes.io/projected/88177313-1d69-4b29-8351-cc7ec39a2e1b-kube-api-access-lwwdh\") pod \"test-curl-7802-140618635221296\" (UID: \"88177313-1d69-4b29-8351-cc7ec39a2e1b\") " pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:47.001835 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:47.001799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:47.157250 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:47.157222 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:47.159276 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:06:47.159251 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88177313_1d69_4b29_8351_cc7ec39a2e1b.slice/crio-9833aacb23b44e7394245f254688ca2cb3530d0dfbe38b7e11ba1df7f823e48a WatchSource:0}: Error finding container 9833aacb23b44e7394245f254688ca2cb3530d0dfbe38b7e11ba1df7f823e48a: Status 404 returned error can't find the container with id 9833aacb23b44e7394245f254688ca2cb3530d0dfbe38b7e11ba1df7f823e48a Jul 29 08:06:47.311689 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:47.311658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618635221296" event={"ID":"88177313-1d69-4b29-8351-cc7ec39a2e1b","Type":"ContainerStarted","Data":"9833aacb23b44e7394245f254688ca2cb3530d0dfbe38b7e11ba1df7f823e48a"} Jul 29 08:06:47.312886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:47.312856 2568 scope.go:117] "RemoveContainer" containerID="12cf1504f701b85bf913502467fb66b57dbf142ae5ba45d8245430bdaa7d4362" Jul 29 08:06:47.312886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:47.312872 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:48.320748 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:48.320711 2568 generic.go:356] "Generic (PLEG): container finished" podID="88177313-1d69-4b29-8351-cc7ec39a2e1b" containerID="1f8ea9b41b62568f3dc2c55f13ab5605bc467b3e9c728d185617540777986f03" exitCode=0 Jul 29 08:06:48.321172 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:48.320804 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618635221296" event={"ID":"88177313-1d69-4b29-8351-cc7ec39a2e1b","Type":"ContainerDied","Data":"1f8ea9b41b62568f3dc2c55f13ab5605bc467b3e9c728d185617540777986f03"} Jul 29 08:06:49.455770 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.455742 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:49.486070 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.486021 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7802-140618635221296_88177313-1d69-4b29-8351-cc7ec39a2e1b/test-curl-7802-140618635221296/0.log" Jul 29 08:06:49.518346 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.518312 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:49.526052 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.526014 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7802-140618635221296"] Jul 29 08:06:49.615908 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.615822 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwwdh\" (UniqueName: \"kubernetes.io/projected/88177313-1d69-4b29-8351-cc7ec39a2e1b-kube-api-access-lwwdh\") pod \"88177313-1d69-4b29-8351-cc7ec39a2e1b\" (UID: \"88177313-1d69-4b29-8351-cc7ec39a2e1b\") " Jul 29 08:06:49.618077 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.618051 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88177313-1d69-4b29-8351-cc7ec39a2e1b-kube-api-access-lwwdh" (OuterVolumeSpecName: "kube-api-access-lwwdh") pod "88177313-1d69-4b29-8351-cc7ec39a2e1b" (UID: "88177313-1d69-4b29-8351-cc7ec39a2e1b"). InnerVolumeSpecName "kube-api-access-lwwdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:06:49.675130 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.675100 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7802-140618635222304"] Jul 29 08:06:49.675504 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.675492 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88177313-1d69-4b29-8351-cc7ec39a2e1b" containerName="test-curl-7802-140618635221296" Jul 29 08:06:49.675553 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.675506 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="88177313-1d69-4b29-8351-cc7ec39a2e1b" containerName="test-curl-7802-140618635221296" Jul 29 08:06:49.675588 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.675573 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="88177313-1d69-4b29-8351-cc7ec39a2e1b" containerName="test-curl-7802-140618635221296" Jul 29 08:06:49.678582 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.678566 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:49.688376 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.688356 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618635222304"] Jul 29 08:06:49.717354 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.717327 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwdh\" (UniqueName: \"kubernetes.io/projected/88177313-1d69-4b29-8351-cc7ec39a2e1b-kube-api-access-lwwdh\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:06:49.818256 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.818213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqqg\" (UniqueName: \"kubernetes.io/projected/28094005-dc08-421d-a7c2-3fadfcea39bd-kube-api-access-jzqqg\") pod \"test-curl-7802-140618635222304\" (UID: \"28094005-dc08-421d-a7c2-3fadfcea39bd\") " pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:49.919753 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.919725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqg\" (UniqueName: \"kubernetes.io/projected/28094005-dc08-421d-a7c2-3fadfcea39bd-kube-api-access-jzqqg\") pod \"test-curl-7802-140618635222304\" (UID: \"28094005-dc08-421d-a7c2-3fadfcea39bd\") " pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:49.931353 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.931327 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqqg\" (UniqueName: \"kubernetes.io/projected/28094005-dc08-421d-a7c2-3fadfcea39bd-kube-api-access-jzqqg\") pod \"test-curl-7802-140618635222304\" (UID: \"28094005-dc08-421d-a7c2-3fadfcea39bd\") " pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:49.988015 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:49.987987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:50.134812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:50.134784 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7802-140618635222304"] Jul 29 08:06:50.136054 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:06:50.136014 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28094005_dc08_421d_a7c2_3fadfcea39bd.slice/crio-36173b82ebdfa4c0bffacf864b06e2ac1593b92e4bf0826cbf52ad4eca93c968 WatchSource:0}: Error finding container 36173b82ebdfa4c0bffacf864b06e2ac1593b92e4bf0826cbf52ad4eca93c968: Status 404 returned error can't find the container with id 36173b82ebdfa4c0bffacf864b06e2ac1593b92e4bf0826cbf52ad4eca93c968 Jul 29 08:06:50.334730 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:50.334689 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618635222304" event={"ID":"28094005-dc08-421d-a7c2-3fadfcea39bd","Type":"ContainerStarted","Data":"36173b82ebdfa4c0bffacf864b06e2ac1593b92e4bf0826cbf52ad4eca93c968"} Jul 29 08:06:50.335912 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:50.335892 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9833aacb23b44e7394245f254688ca2cb3530d0dfbe38b7e11ba1df7f823e48a" Jul 29 08:06:50.335912 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:50.335896 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:50.349157 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:06:50.349130 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7802-140618635221296\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="88177313-1d69-4b29-8351-cc7ec39a2e1b" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:50.737519 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:06:50.737428 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7802-140618635221296\" is forbidden: User \"system:node:ip-10-0-136-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-73.ec2.internal' and this object" podUID="88177313-1d69-4b29-8351-cc7ec39a2e1b" pod="opendatahub/test-curl-7802-140618635221296" Jul 29 08:06:50.738937 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:50.738914 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88177313-1d69-4b29-8351-cc7ec39a2e1b" path="/var/lib/kubelet/pods/88177313-1d69-4b29-8351-cc7ec39a2e1b/volumes" Jul 29 08:06:51.350207 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:51.342325 2568 generic.go:356] "Generic (PLEG): container finished" podID="28094005-dc08-421d-a7c2-3fadfcea39bd" containerID="f717966c6cf1c7555088b874779d241a236a65424b5c65a5903742d8f6888e2c" exitCode=0 Jul 29 08:06:51.350207 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:51.342410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7802-140618635222304" event={"ID":"28094005-dc08-421d-a7c2-3fadfcea39bd","Type":"ContainerDied","Data":"f717966c6cf1c7555088b874779d241a236a65424b5c65a5903742d8f6888e2c"} Jul 29 08:06:52.488731 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.488704 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:52.512865 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.512837 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7802-140618635222304_28094005-dc08-421d-a7c2-3fadfcea39bd/test-curl-7802-140618635222304/0.log" Jul 29 08:06:52.543766 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.543732 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7802-140618635222304"] Jul 29 08:06:52.550222 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.550196 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7802-140618635222304"] Jul 29 08:06:52.645406 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.645339 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzqqg\" (UniqueName: \"kubernetes.io/projected/28094005-dc08-421d-a7c2-3fadfcea39bd-kube-api-access-jzqqg\") pod \"28094005-dc08-421d-a7c2-3fadfcea39bd\" (UID: \"28094005-dc08-421d-a7c2-3fadfcea39bd\") " Jul 29 08:06:52.647489 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.647459 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28094005-dc08-421d-a7c2-3fadfcea39bd-kube-api-access-jzqqg" (OuterVolumeSpecName: "kube-api-access-jzqqg") pod "28094005-dc08-421d-a7c2-3fadfcea39bd" (UID: "28094005-dc08-421d-a7c2-3fadfcea39bd"). InnerVolumeSpecName "kube-api-access-jzqqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:06:52.738068 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.738006 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28094005-dc08-421d-a7c2-3fadfcea39bd" path="/var/lib/kubelet/pods/28094005-dc08-421d-a7c2-3fadfcea39bd/volumes" Jul 29 08:06:52.745943 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:52.745923 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzqqg\" (UniqueName: \"kubernetes.io/projected/28094005-dc08-421d-a7c2-3fadfcea39bd-kube-api-access-jzqqg\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:06:53.353004 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:53.352972 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7802-140618635222304" Jul 29 08:06:53.353214 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:53.353076 2568 scope.go:117] "RemoveContainer" containerID="f717966c6cf1c7555088b874779d241a236a65424b5c65a5903742d8f6888e2c" Jul 29 08:06:56.221693 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.221653 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:06:56.740149 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.740114 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf"] Jul 29 08:06:56.740525 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.740513 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28094005-dc08-421d-a7c2-3fadfcea39bd" containerName="test-curl-7802-140618635222304" Jul 29 08:06:56.740574 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.740527 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="28094005-dc08-421d-a7c2-3fadfcea39bd" containerName="test-curl-7802-140618635222304" Jul 29 08:06:56.740610 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.740588 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="28094005-dc08-421d-a7c2-3fadfcea39bd" containerName="test-curl-7802-140618635222304" Jul 29 08:06:56.745160 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.745138 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.749472 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.749452 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-940406-openshift-default-dockercfg-qx7pt\"" Jul 29 08:06:56.772441 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.772407 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf"] Jul 29 08:06:56.881976 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.881942 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istiod-ca-cert\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882190 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-token\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882190 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-certs\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882190 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882097 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjpl\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-kube-api-access-pvjpl\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882190 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-credential-socket\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882190 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882149 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-envoy\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882445 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-data\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882445 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882327 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-podinfo\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.882445 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.882414 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-socket\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983563 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-data\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983563 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983563 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-podinfo\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983831 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-socket\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983831 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istiod-ca-cert\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983831 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-token\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983831 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-certs\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983831 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjpl\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-kube-api-access-pvjpl\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.983831 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-credential-socket\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.984199 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-envoy\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.984199 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.983994 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-data\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.984199 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.984184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-socket\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.984349 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.984218 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-credential-socket\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.984521 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.984461 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-certs\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.984843 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.984726 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istiod-ca-cert\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.986402 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.986380 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-podinfo\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.986517 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.986455 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-envoy\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.996477 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.996414 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-token\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:56.998525 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:56.998506 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjpl\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-kube-api-access-pvjpl\") pod \"e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:57.055793 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.055762 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:57.208072 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.208029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf"] Jul 29 08:06:57.210461 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:06:57.210433 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88206e4_4cb8_43ac_a2dd_5012efa6c85d.slice/crio-faf507a20765a36d64c0fdf4bab1fbe301b7b12613c88bb18c00bd618f5ebed8 WatchSource:0}: Error finding container faf507a20765a36d64c0fdf4bab1fbe301b7b12613c88bb18c00bd618f5ebed8: Status 404 returned error can't find the container with id faf507a20765a36d64c0fdf4bab1fbe301b7b12613c88bb18c00bd618f5ebed8 Jul 29 08:06:57.212623 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.212591 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:06:57.212705 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.212654 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:06:57.212705 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.212682 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:06:57.376494 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.376461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" event={"ID":"b88206e4-4cb8-43ac-a2dd-5012efa6c85d","Type":"ContainerStarted","Data":"80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e"} Jul 29 08:06:57.376960 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.376515 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" event={"ID":"b88206e4-4cb8-43ac-a2dd-5012efa6c85d","Type":"ContainerStarted","Data":"faf507a20765a36d64c0fdf4bab1fbe301b7b12613c88bb18c00bd618f5ebed8"} Jul 29 08:06:57.421860 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:57.421744 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podStartSLOduration=1.4217233980000001 podStartE2EDuration="1.421723398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:06:57.416758705 +0000 UTC m=+3127.188209505" watchObservedRunningTime="2026-07-29 08:06:57.421723398 +0000 UTC m=+3127.193174199" Jul 29 08:06:58.056143 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:58.056108 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:58.061272 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:58.061248 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:58.382227 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:58.382146 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:58.383209 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:58.383183 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:06:58.778175 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:58.778143 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:06:59.660282 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:59.660245 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:06:59.764453 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:59.764415 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:06:59.965225 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:06:59.965147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:00.601464 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:00.601431 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:00.921887 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:00.921858 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:01.010500 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:01.010460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:06.938668 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:06.938631 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:07.038836 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:07.038801 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:14.424289 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:14.424244 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 08:07:14.425014 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:14.424334 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 08:07:14.440164 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:14.440126 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:15.370622 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.370583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:15.626763 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.626657 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg"] Jul 29 08:07:15.630659 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.630635 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.633748 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.633722 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f1dc98-openshift-default-dockercfg-x2vsc\"" Jul 29 08:07:15.643786 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.643758 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg"] Jul 29 08:07:15.658308 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.658274 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:15.770344 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-data\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770344 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-socket\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770613 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24m8r\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-kube-api-access-24m8r\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770613 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-certs\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770613 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-envoy\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770770 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770601 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-credential-socket\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770770 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-podinfo\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770770 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770687 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-token\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.770770 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.770732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b836a79e-dc47-4f1a-8461-be8412776e3b-istiod-ca-cert\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871492 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-envoy\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-credential-socket\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-podinfo\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-token\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871931 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b836a79e-dc47-4f1a-8461-be8412776e3b-istiod-ca-cert\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871931 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-data\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871931 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871896 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-credential-socket\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.871931 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871908 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-socket\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.872247 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.871949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24m8r\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-kube-api-access-24m8r\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.872247 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.872034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-certs\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.872247 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.872211 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-socket\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.872540 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.872513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-data\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.872695 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.872545 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b836a79e-dc47-4f1a-8461-be8412776e3b-istiod-ca-cert\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.872904 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.872858 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-certs\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.874937 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.874914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-envoy\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.875064 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.875022 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-podinfo\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.881173 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.881114 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24m8r\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-kube-api-access-24m8r\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.881283 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.881258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-token\") pod \"e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:15.915186 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.915153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:15.945585 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:15.945549 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:16.105592 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.104975 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg"] Jul 29 08:07:16.110297 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:07:16.110265 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb836a79e_dc47_4f1a_8461_be8412776e3b.slice/crio-52455261d4f8f8155ffa577fa9fb103d49fe2163a02b75bb641ec80cb9ed1389 WatchSource:0}: Error finding container 52455261d4f8f8155ffa577fa9fb103d49fe2163a02b75bb641ec80cb9ed1389: Status 404 returned error can't find the container with id 52455261d4f8f8155ffa577fa9fb103d49fe2163a02b75bb641ec80cb9ed1389 Jul 29 08:07:16.112369 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.112333 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:16.112473 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.112401 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:16.112473 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.112430 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:16.489334 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.489297 2568 generic.go:356] "Generic (PLEG): container finished" podID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerID="80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e" exitCode=0 Jul 29 08:07:16.489543 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.489381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" event={"ID":"b88206e4-4cb8-43ac-a2dd-5012efa6c85d","Type":"ContainerDied","Data":"80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e"} Jul 29 08:07:16.489911 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.489892 2568 scope.go:117] "RemoveContainer" containerID="80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e" Jul 29 08:07:16.491115 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.491088 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:16.491221 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.491157 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:16.491221 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.491094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" event={"ID":"b836a79e-dc47-4f1a-8461-be8412776e3b","Type":"ContainerStarted","Data":"a8d755f0977834dc5842ed6027f0e50a90f818fbe4bed04a78970edaeecdf713"} Jul 29 08:07:16.491337 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.491233 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" event={"ID":"b836a79e-dc47-4f1a-8461-be8412776e3b","Type":"ContainerStarted","Data":"52455261d4f8f8155ffa577fa9fb103d49fe2163a02b75bb641ec80cb9ed1389"} Jul 29 08:07:16.491337 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.491190 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:16.561667 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.561613 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podStartSLOduration=1.561590356 podStartE2EDuration="1.561590356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:07:16.560080308 +0000 UTC m=+3146.331531100" watchObservedRunningTime="2026-07-29 08:07:16.561590356 +0000 UTC m=+3146.333041156" Jul 29 08:07:16.946480 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:16.946445 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:17.498889 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:17.498850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" event={"ID":"b88206e4-4cb8-43ac-a2dd-5012efa6c85d","Type":"ContainerStarted","Data":"c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e"} Jul 29 08:07:17.953143 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:17.953111 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:18.056133 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.056097 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:07:18.056416 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.056389 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Jul 29 08:07:18.056463 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.056441 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Jul 29 08:07:18.504749 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.504717 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:18.505678 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.505657 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:18.812063 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.811971 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:18.914665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:18.914630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:19.057161 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:19.057127 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Jul 29 08:07:19.057589 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:19.057185 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Jul 29 08:07:20.056308 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:20.056272 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Jul 29 08:07:20.056475 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:20.056327 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Jul 29 08:07:21.049838 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:21.049792 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:21.057151 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:21.057108 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Jul 29 08:07:21.057314 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:21.057167 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Jul 29 08:07:21.480195 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:21.480150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:21.577700 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:21.577665 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:21.655167 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:21.655133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:22.060208 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:22.060176 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:07:22.060713 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:22.060686 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:07:22.062239 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:22.062219 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:07:38.620485 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.620380 2568 generic.go:356] "Generic (PLEG): container finished" podID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerID="a8d755f0977834dc5842ed6027f0e50a90f818fbe4bed04a78970edaeecdf713" exitCode=0 Jul 29 08:07:38.620485 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.620468 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" event={"ID":"b836a79e-dc47-4f1a-8461-be8412776e3b","Type":"ContainerDied","Data":"a8d755f0977834dc5842ed6027f0e50a90f818fbe4bed04a78970edaeecdf713"} Jul 29 08:07:38.620997 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.620850 2568 scope.go:117] "RemoveContainer" containerID="a8d755f0977834dc5842ed6027f0e50a90f818fbe4bed04a78970edaeecdf713" Jul 29 08:07:38.621946 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.621920 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:38.622019 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.622005 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:38.622098 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.622075 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:07:38.738886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:38.738797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:07:39.628488 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:39.628450 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" event={"ID":"b836a79e-dc47-4f1a-8461-be8412776e3b","Type":"ContainerStarted","Data":"ca42ed06a93d8eccacadbfc3ace1113387d812a7f2a459513fa1a931144e4e7f"} Jul 29 08:07:39.945950 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:39.945922 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:39.946151 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:39.946131 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Jul 29 08:07:39.946210 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:39.946176 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Jul 29 08:07:40.946879 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:40.946815 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Jul 29 08:07:40.947455 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:40.946886 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Jul 29 08:07:41.945925 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:41.945888 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Jul 29 08:07:41.946257 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:41.946225 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Jul 29 08:07:42.946129 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:42.946090 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Jul 29 08:07:42.946621 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:42.946170 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Jul 29 08:07:43.946065 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:43.946007 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Jul 29 08:07:43.946241 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:43.946096 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Jul 29 08:07:44.946793 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:44.946751 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Jul 29 08:07:44.947229 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:44.946815 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Jul 29 08:07:45.950307 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:45.950278 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:45.950781 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:45.950614 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:45.951223 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:45.951204 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:07:50.081828 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:07:50.081791 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:39.339246 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.339210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:39.544427 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.544392 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt"] Jul 29 08:08:39.548103 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.548085 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.551453 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.551436 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f73166-openshift-default-dockercfg-484gl\"" Jul 29 08:08:39.567480 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.567452 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt"] Jul 29 08:08:39.585722 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.585691 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:39.603404 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-credential-socket\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603404 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-envoy\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603540 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603472 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-token\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603540 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-podinfo\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603609 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603609 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7sq\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-kube-api-access-pj7sq\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603679 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-socket\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603679 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-data\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.603743 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.603686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-certs\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.704948 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.704903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-socket\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.704948 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.704954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-data\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.704993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-certs\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-credential-socket\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-envoy\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-token\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-podinfo\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705573 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7sq\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-kube-api-access-pj7sq\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705573 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705428 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-socket\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705573 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-data\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705573 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705497 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-certs\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.705769 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.705681 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-credential-socket\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.706023 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.706000 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.707716 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.707679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-envoy\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.707995 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.707974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-podinfo\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.713960 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.713938 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-token\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.714386 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.714364 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7sq\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-kube-api-access-pj7sq\") pod \"e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:39.859810 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:39.859728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:40.018796 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.018764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt"] Jul 29 08:08:40.020197 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:08:40.020168 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf848a6_d78a_4378_80c1_6cb0231ec9f3.slice/crio-6c5cb22fe7348a3ed2b969518c5395b77af0901f2f8cd3563569c2cfa56833d1 WatchSource:0}: Error finding container 6c5cb22fe7348a3ed2b969518c5395b77af0901f2f8cd3563569c2cfa56833d1: Status 404 returned error can't find the container with id 6c5cb22fe7348a3ed2b969518c5395b77af0901f2f8cd3563569c2cfa56833d1 Jul 29 08:08:40.022662 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.022627 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:08:40.022764 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.022729 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:08:40.022829 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.022778 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:08:40.150882 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.150779 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:40.980395 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.980362 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" event={"ID":"4bf848a6-d78a-4378-80c1-6cb0231ec9f3","Type":"ContainerStarted","Data":"e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884"} Jul 29 08:08:40.980790 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:40.980416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" event={"ID":"4bf848a6-d78a-4378-80c1-6cb0231ec9f3","Type":"ContainerStarted","Data":"6c5cb22fe7348a3ed2b969518c5395b77af0901f2f8cd3563569c2cfa56833d1"} Jul 29 08:08:41.003887 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:41.003836 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podStartSLOduration=2.003820992 podStartE2EDuration="2.003820992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:08:41.001753688 +0000 UTC m=+3230.773204488" watchObservedRunningTime="2026-07-29 08:08:41.003820992 +0000 UTC m=+3230.775271791" Jul 29 08:08:41.860711 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:41.860678 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:41.866164 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:41.866137 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:41.986706 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:41.986668 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:41.987651 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:41.987629 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:08:42.977165 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:42.977106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:43.070132 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:43.070102 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:44.897641 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:44.897599 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:44.998241 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:44.998210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:57.821162 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:57.821123 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:08:58.684134 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.684095 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt"] Jul 29 08:08:58.689258 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.689241 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:08:58.692499 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.692474 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 08:08:58.698766 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.698740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt"] Jul 29 08:08:58.800609 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.800568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff038a6d-63ee-4137-99e3-5f81e3eb41c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt\" (UID: \"ff038a6d-63ee-4137-99e3-5f81e3eb41c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:08:58.901497 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.901465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff038a6d-63ee-4137-99e3-5f81e3eb41c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt\" (UID: \"ff038a6d-63ee-4137-99e3-5f81e3eb41c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:08:58.902110 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:58.902085 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff038a6d-63ee-4137-99e3-5f81e3eb41c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt\" (UID: \"ff038a6d-63ee-4137-99e3-5f81e3eb41c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:08:59.002745 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:59.002666 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:08:59.167238 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:08:59.167210 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt"] Jul 29 08:09:00.092514 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:00.092482 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff038a6d-63ee-4137-99e3-5f81e3eb41c3" containerID="4fa8bcc80b5e4019c4677515c1075dc7e8f68f476d16a46bfc6db133a6b9a2ac" exitCode=0 Jul 29 08:09:00.092930 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:00.092577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" event={"ID":"ff038a6d-63ee-4137-99e3-5f81e3eb41c3","Type":"ContainerDied","Data":"4fa8bcc80b5e4019c4677515c1075dc7e8f68f476d16a46bfc6db133a6b9a2ac"} Jul 29 08:09:00.092930 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:00.092665 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" event={"ID":"ff038a6d-63ee-4137-99e3-5f81e3eb41c3","Type":"ContainerStarted","Data":"1d89cbd735e78c0dbfa94c74c943a1b40f42d151d044fd018834798f9b1e2b24"} Jul 29 08:09:01.099891 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.099854 2568 generic.go:356] "Generic (PLEG): container finished" podID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerID="e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884" exitCode=0 Jul 29 08:09:01.100386 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.099926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" event={"ID":"4bf848a6-d78a-4378-80c1-6cb0231ec9f3","Type":"ContainerDied","Data":"e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884"} Jul 29 08:09:01.100719 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.100451 2568 scope.go:117] "RemoveContainer" containerID="e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884" Jul 29 08:09:01.101590 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.101559 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:09:01.101703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.101628 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:09:01.101703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.101665 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 29 08:09:01.271291 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.271261 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:09:01.323697 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.323656 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff038a6d-63ee-4137-99e3-5f81e3eb41c3-maas-api-service-ca\") pod \"ff038a6d-63ee-4137-99e3-5f81e3eb41c3\" (UID: \"ff038a6d-63ee-4137-99e3-5f81e3eb41c3\") " Jul 29 08:09:01.324109 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.324083 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff038a6d-63ee-4137-99e3-5f81e3eb41c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff038a6d-63ee-4137-99e3-5f81e3eb41c3" (UID: "ff038a6d-63ee-4137-99e3-5f81e3eb41c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:09:01.424881 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:01.424845 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff038a6d-63ee-4137-99e3-5f81e3eb41c3-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:02.107303 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.107269 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" Jul 29 08:09:02.107776 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.107300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt" event={"ID":"ff038a6d-63ee-4137-99e3-5f81e3eb41c3","Type":"ContainerDied","Data":"1d89cbd735e78c0dbfa94c74c943a1b40f42d151d044fd018834798f9b1e2b24"} Jul 29 08:09:02.107776 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.107354 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d89cbd735e78c0dbfa94c74c943a1b40f42d151d044fd018834798f9b1e2b24" Jul 29 08:09:02.109418 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.109394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" event={"ID":"4bf848a6-d78a-4378-80c1-6cb0231ec9f3","Type":"ContainerStarted","Data":"1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880"} Jul 29 08:09:02.860783 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.860743 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:09:02.861118 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.861093 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Jul 29 08:09:02.861191 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:02.861151 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Jul 29 08:09:03.860404 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:03.860362 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Jul 29 08:09:03.860798 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:03.860431 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Jul 29 08:09:04.860330 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:04.860288 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Jul 29 08:09:04.860506 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:04.860361 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Jul 29 08:09:05.860770 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:05.860733 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Jul 29 08:09:05.861173 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:05.860792 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Jul 29 08:09:06.860232 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:06.860190 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Jul 29 08:09:06.860427 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:06.860262 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Jul 29 08:09:07.757267 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:07.757236 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt"] Jul 29 08:09:07.760895 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:07.760868 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f73166-gtdmt"] Jul 29 08:09:07.864140 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:07.864111 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:09:07.864483 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:07.864466 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:09:07.865056 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:07.865021 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:09:08.280245 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:08.280213 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:08.340983 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:08.340948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:08.738348 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:08.738317 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff038a6d-63ee-4137-99e3-5f81e3eb41c3" path="/var/lib/kubelet/pods/ff038a6d-63ee-4137-99e3-5f81e3eb41c3/volumes" Jul 29 08:09:14.256081 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:14.256027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:14.665384 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:14.665346 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt"] Jul 29 08:09:14.665586 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:14.665564 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" containerID="cri-o://1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880" gracePeriod=30 Jul 29 08:09:15.374282 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:15.374249 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:15.471932 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:15.471891 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:19.933087 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:19.932996 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:09:20.011400 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011369 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-credential-socket\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011599 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011425 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-socket\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011599 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011458 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-token\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011599 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011474 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-certs\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011599 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011508 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-envoy\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011599 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011525 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj7sq\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-kube-api-access-pj7sq\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011607 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istiod-ca-cert\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011677 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:09:20.011886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011685 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-podinfo\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011770 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-data\") pod \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\" (UID: \"4bf848a6-d78a-4378-80c1-6cb0231ec9f3\") " Jul 29 08:09:20.011886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011801 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:09:20.011886 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.011861 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:09:20.012228 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.012158 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.012228 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.012178 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.012228 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.012192 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.012228 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.012191 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:09:20.012635 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.012616 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:09:20.014076 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.014031 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:09:20.014076 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.014057 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-kube-api-access-pj7sq" (OuterVolumeSpecName: "kube-api-access-pj7sq") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "kube-api-access-pj7sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:09:20.014341 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.014315 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 08:09:20.014428 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.014402 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4bf848a6-d78a-4378-80c1-6cb0231ec9f3" (UID: "4bf848a6-d78a-4378-80c1-6cb0231ec9f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:09:20.113721 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.113686 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.113721 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.113715 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.113721 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.113725 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj7sq\" (UniqueName: \"kubernetes.io/projected/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-kube-api-access-pj7sq\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.113956 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.113734 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.113956 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.113743 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.113956 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.113750 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf848a6-d78a-4378-80c1-6cb0231ec9f3-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:09:20.217822 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.217731 2568 generic.go:356] "Generic (PLEG): container finished" podID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerID="1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880" exitCode=0 Jul 29 08:09:20.217822 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.217786 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" Jul 29 08:09:20.218001 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.217822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" event={"ID":"4bf848a6-d78a-4378-80c1-6cb0231ec9f3","Type":"ContainerDied","Data":"1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880"} Jul 29 08:09:20.218001 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.217881 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt" event={"ID":"4bf848a6-d78a-4378-80c1-6cb0231ec9f3","Type":"ContainerDied","Data":"6c5cb22fe7348a3ed2b969518c5395b77af0901f2f8cd3563569c2cfa56833d1"} Jul 29 08:09:20.218001 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.217912 2568 scope.go:117] "RemoveContainer" containerID="1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880" Jul 29 08:09:20.228939 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.228917 2568 scope.go:117] "RemoveContainer" containerID="e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884" Jul 29 08:09:20.237963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.237944 2568 scope.go:117] "RemoveContainer" containerID="1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880" Jul 29 08:09:20.238234 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:09:20.238218 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880\": container with ID starting with 1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880 not found: ID does not exist" containerID="1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880" Jul 29 08:09:20.238292 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.238243 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880"} err="failed to get container status \"1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880\": rpc error: code = NotFound desc = could not find container \"1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880\": container with ID starting with 1ebd7bd09dbd7e9701c6fb08afd12de9b65f01a0fe55f14daab5a19353009880 not found: ID does not exist" Jul 29 08:09:20.238292 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.238259 2568 scope.go:117] "RemoveContainer" containerID="e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884" Jul 29 08:09:20.238457 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:09:20.238443 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884\": container with ID starting with e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884 not found: ID does not exist" containerID="e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884" Jul 29 08:09:20.238497 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.238463 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884"} err="failed to get container status \"e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884\": rpc error: code = NotFound desc = could not find container \"e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884\": container with ID starting with e65c62d81769b46bfafa6e8ef735254b59cc369b55e73dea3cc170e90c359884 not found: ID does not exist" Jul 29 08:09:20.251796 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.251768 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt"] Jul 29 08:09:20.263805 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.263778 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73166-openshift-default-79b447d6c4-bl8rt"] Jul 29 08:09:20.738221 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:20.738188 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" path="/var/lib/kubelet/pods/4bf848a6-d78a-4378-80c1-6cb0231ec9f3/volumes" Jul 29 08:09:22.549366 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:22.549330 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:22.641305 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:22.641260 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:38.970589 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:38.970531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:39.074516 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:39.074484 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:41.873371 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:41.873341 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:45.972530 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:45.972492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:09:50.975966 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:50.975938 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 08:09:50.985588 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:09:50.985566 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 08:10:10.477567 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:10.477528 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:20.483501 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:20.483417 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:20.620849 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:20.620816 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:27.513176 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:27.513146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:27.616525 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:27.616496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:27.866261 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:27.866177 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf"] Jul 29 08:10:27.866445 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:27.866423 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" containerID="cri-o://c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e" gracePeriod=30 Jul 29 08:10:27.922709 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:27.922680 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:28.152003 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:28.151917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:33.123921 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.123898 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:10:33.194520 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194495 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-socket\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istiod-ca-cert\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194586 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-credential-socket\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194607 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-envoy\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194647 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjpl\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-kube-api-access-pvjpl\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194664 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-podinfo\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194923 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194682 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-certs\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194923 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194758 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:33.194923 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194821 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-token\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194923 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194860 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-data\") pod \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\" (UID: \"b88206e4-4cb8-43ac-a2dd-5012efa6c85d\") " Jul 29 08:10:33.194923 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.194910 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:33.195207 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:33.195207 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195071 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:33.195289 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195259 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.195289 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195278 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.195393 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195292 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.195393 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195305 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.195724 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.195642 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-data" (OuterVolumeSpecName: "istio-data") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:33.196940 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.196919 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:33.197230 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.197211 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-token" (OuterVolumeSpecName: "istio-token") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:33.197465 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.197435 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 08:10:33.197559 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.197442 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-kube-api-access-pvjpl" (OuterVolumeSpecName: "kube-api-access-pvjpl") pod "b88206e4-4cb8-43ac-a2dd-5012efa6c85d" (UID: "b88206e4-4cb8-43ac-a2dd-5012efa6c85d"). InnerVolumeSpecName "kube-api-access-pvjpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:33.296438 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.296405 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.296438 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.296433 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.296438 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.296442 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.296673 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.296451 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjpl\" (UniqueName: \"kubernetes.io/projected/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-kube-api-access-pvjpl\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.296673 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.296460 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88206e4-4cb8-43ac-a2dd-5012efa6c85d-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:33.638582 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.638494 2568 generic.go:356] "Generic (PLEG): container finished" podID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerID="c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e" exitCode=0 Jul 29 08:10:33.638582 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.638547 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" Jul 29 08:10:33.638808 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.638548 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" event={"ID":"b88206e4-4cb8-43ac-a2dd-5012efa6c85d","Type":"ContainerDied","Data":"c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e"} Jul 29 08:10:33.638808 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.638682 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf" event={"ID":"b88206e4-4cb8-43ac-a2dd-5012efa6c85d","Type":"ContainerDied","Data":"faf507a20765a36d64c0fdf4bab1fbe301b7b12613c88bb18c00bd618f5ebed8"} Jul 29 08:10:33.638808 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.638712 2568 scope.go:117] "RemoveContainer" containerID="c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e" Jul 29 08:10:33.648884 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.648866 2568 scope.go:117] "RemoveContainer" containerID="80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e" Jul 29 08:10:33.657872 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.657857 2568 scope.go:117] "RemoveContainer" containerID="c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e" Jul 29 08:10:33.658229 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:10:33.658211 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e\": container with ID starting with c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e not found: ID does not exist" containerID="c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e" Jul 29 08:10:33.658308 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.658235 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e"} err="failed to get container status \"c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e\": rpc error: code = NotFound desc = could not find container \"c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e\": container with ID starting with c4ae1fe12d5a4057edac4b5bdde7f99e6f2fb648e18858fd39a0300f80afc28e not found: ID does not exist" Jul 29 08:10:33.658308 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.658251 2568 scope.go:117] "RemoveContainer" containerID="80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e" Jul 29 08:10:33.658498 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:10:33.658480 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e\": container with ID starting with 80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e not found: ID does not exist" containerID="80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e" Jul 29 08:10:33.658555 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.658508 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e"} err="failed to get container status \"80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e\": rpc error: code = NotFound desc = could not find container \"80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e\": container with ID starting with 80fe21c127dfc21c481d4a80dda2064557a472383096be94fd01c68be48d7d0e not found: ID does not exist" Jul 29 08:10:33.667005 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.666982 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf"] Jul 29 08:10:33.669202 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:33.669183 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-940406-openshift-default-664454b9c6-w2vqf"] Jul 29 08:10:34.738450 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:34.738417 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" path="/var/lib/kubelet/pods/b88206e4-4cb8-43ac-a2dd-5012efa6c85d/volumes" Jul 29 08:10:35.516076 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:35.516028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:36.782395 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:36.782363 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:38.061775 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:38.061746 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:42.269500 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:42.269464 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:42.377585 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:42.377553 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:42.569708 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:42.569638 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg"] Jul 29 08:10:42.569846 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:42.569818 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" containerID="cri-o://ca42ed06a93d8eccacadbfc3ace1113387d812a7f2a459513fa1a931144e4e7f" gracePeriod=30 Jul 29 08:10:42.594249 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:42.594219 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:42.901312 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:42.901228 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:45.953724 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:45.953680 2568 patch_prober.go:28] interesting pod/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 08:10:45.954109 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:45.953751 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 08:10:47.725383 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.725354 2568 generic.go:356] "Generic (PLEG): container finished" podID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerID="ca42ed06a93d8eccacadbfc3ace1113387d812a7f2a459513fa1a931144e4e7f" exitCode=0 Jul 29 08:10:47.725757 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.725428 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" event={"ID":"b836a79e-dc47-4f1a-8461-be8412776e3b","Type":"ContainerDied","Data":"ca42ed06a93d8eccacadbfc3ace1113387d812a7f2a459513fa1a931144e4e7f"} Jul 29 08:10:47.725757 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.725495 2568 scope.go:117] "RemoveContainer" containerID="a8d755f0977834dc5842ed6027f0e50a90f818fbe4bed04a78970edaeecdf713" Jul 29 08:10:47.839552 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.839485 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:10:47.928609 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928583 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-envoy\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.928806 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928629 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-certs\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.928806 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928650 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-socket\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.928806 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928716 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-podinfo\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.928806 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928742 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-credential-socket\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.928806 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928782 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-token\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.929112 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928808 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b836a79e-dc47-4f1a-8461-be8412776e3b-istiod-ca-cert\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.929112 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928862 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-data\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.929112 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928886 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24m8r\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-kube-api-access-24m8r\") pod \"b836a79e-dc47-4f1a-8461-be8412776e3b\" (UID: \"b836a79e-dc47-4f1a-8461-be8412776e3b\") " Jul 29 08:10:47.929112 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.928906 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:47.929112 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.929013 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:47.929401 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.929310 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:47.929401 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.929329 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:47.929506 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.929366 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:47.929742 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.929704 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b836a79e-dc47-4f1a-8461-be8412776e3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:47.930004 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.929968 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:47.931133 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.931109 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:10:47.931368 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.931346 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-kube-api-access-24m8r" (OuterVolumeSpecName: "kube-api-access-24m8r") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "kube-api-access-24m8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:47.931678 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.931665 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:47.931775 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:47.931752 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b836a79e-dc47-4f1a-8461-be8412776e3b" (UID: "b836a79e-dc47-4f1a-8461-be8412776e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 08:10:48.029862 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029837 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.029862 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029858 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b836a79e-dc47-4f1a-8461-be8412776e3b-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.029862 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029867 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.030114 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029876 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24m8r\" (UniqueName: \"kubernetes.io/projected/b836a79e-dc47-4f1a-8461-be8412776e3b-kube-api-access-24m8r\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.030114 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029886 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.030114 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029894 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b836a79e-dc47-4f1a-8461-be8412776e3b-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.030114 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.029901 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b836a79e-dc47-4f1a-8461-be8412776e3b-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:10:48.733605 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.733573 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" Jul 29 08:10:48.738666 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.738635 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg" event={"ID":"b836a79e-dc47-4f1a-8461-be8412776e3b","Type":"ContainerDied","Data":"52455261d4f8f8155ffa577fa9fb103d49fe2163a02b75bb641ec80cb9ed1389"} Jul 29 08:10:48.738933 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.738690 2568 scope.go:117] "RemoveContainer" containerID="ca42ed06a93d8eccacadbfc3ace1113387d812a7f2a459513fa1a931144e4e7f" Jul 29 08:10:48.760011 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.759988 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg"] Jul 29 08:10:48.764976 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.764952 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1dc98-openshift-default-867769865d-ps4jg"] Jul 29 08:10:48.823880 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.823818 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:48.824117 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.823949 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj"] Jul 29 08:10:48.827081 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.827054 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-984886-29755200-hftgj"] Jul 29 08:10:48.908165 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.908124 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:48.996460 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:48.996381 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:49.138776 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:49.138740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:49.531188 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:49.531158 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:50.739853 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:50.739823 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0325e2e1-f143-44fd-9c8b-f69bafac1024" path="/var/lib/kubelet/pods/0325e2e1-f143-44fd-9c8b-f69bafac1024/volumes" Jul 29 08:10:50.740314 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:50.740299 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" path="/var/lib/kubelet/pods/b836a79e-dc47-4f1a-8461-be8412776e3b/volumes" Jul 29 08:10:51.374366 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:51.374333 2568 scope.go:117] "RemoveContainer" containerID="6a64ea6d9a44a7ca58bfdb88c6570d32cc51bd8a1704da870beb737a92f5164a" Jul 29 08:10:55.708615 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:55.708580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:55.849665 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:55.849631 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:56.014479 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.014376 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:56.443671 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.443635 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn"] Jul 29 08:10:56.444078 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444065 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" Jul 29 08:10:56.444078 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444079 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444095 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444101 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444108 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444113 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444126 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444131 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444137 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff038a6d-63ee-4137-99e3-5f81e3eb41c3" containerName="revoke-keys" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444142 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff038a6d-63ee-4137-99e3-5f81e3eb41c3" containerName="revoke-keys" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444157 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" Jul 29 08:10:56.444197 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444163 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" Jul 29 08:10:56.444531 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444225 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" Jul 29 08:10:56.444531 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444234 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" Jul 29 08:10:56.444531 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444240 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf848a6-d78a-4378-80c1-6cb0231ec9f3" containerName="istio-proxy" Jul 29 08:10:56.444531 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444245 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff038a6d-63ee-4137-99e3-5f81e3eb41c3" containerName="revoke-keys" Jul 29 08:10:56.444531 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444253 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" Jul 29 08:10:56.444531 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.444258 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" Jul 29 08:10:56.448719 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.448693 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:10:56.459788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.452323 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2z4n9\"" Jul 29 08:10:56.461253 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.461224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn"] Jul 29 08:10:56.509144 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.509106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a405563-5624-4dd8-ae28-cc610e6c64b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-71e038-b5whn\" (UID: \"3a405563-5624-4dd8-ae28-cc610e6c64b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:10:56.609741 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.609698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a405563-5624-4dd8-ae28-cc610e6c64b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-71e038-b5whn\" (UID: \"3a405563-5624-4dd8-ae28-cc610e6c64b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:10:56.610384 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.610357 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a405563-5624-4dd8-ae28-cc610e6c64b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-71e038-b5whn\" (UID: \"3a405563-5624-4dd8-ae28-cc610e6c64b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:10:56.767839 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.767756 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:10:56.916732 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:56.916702 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn"] Jul 29 08:10:56.918110 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:10:56.918083 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a405563_5624_4dd8_ae28_cc610e6c64b8.slice/crio-5c6affc9f2644750dae1e56e4592a6a47dcba725b397c07ffc8e126e7c8bed77 WatchSource:0}: Error finding container 5c6affc9f2644750dae1e56e4592a6a47dcba725b397c07ffc8e126e7c8bed77: Status 404 returned error can't find the container with id 5c6affc9f2644750dae1e56e4592a6a47dcba725b397c07ffc8e126e7c8bed77 Jul 29 08:10:57.247352 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:57.247326 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:10:57.786385 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:57.786347 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" event={"ID":"3a405563-5624-4dd8-ae28-cc610e6c64b8","Type":"ContainerStarted","Data":"ac8eeca24e0f5464933965bf6cb99a6ff24c19f8c7ae0135c3f81ce82764c9a4"} Jul 29 08:10:57.786759 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:57.786396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" event={"ID":"3a405563-5624-4dd8-ae28-cc610e6c64b8","Type":"ContainerStarted","Data":"5c6affc9f2644750dae1e56e4592a6a47dcba725b397c07ffc8e126e7c8bed77"} Jul 29 08:10:58.792850 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:58.792821 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a405563-5624-4dd8-ae28-cc610e6c64b8" containerID="ac8eeca24e0f5464933965bf6cb99a6ff24c19f8c7ae0135c3f81ce82764c9a4" exitCode=0 Jul 29 08:10:58.793330 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:58.792907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" event={"ID":"3a405563-5624-4dd8-ae28-cc610e6c64b8","Type":"ContainerDied","Data":"ac8eeca24e0f5464933965bf6cb99a6ff24c19f8c7ae0135c3f81ce82764c9a4"} Jul 29 08:10:59.937064 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:10:59.937024 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:11:00.040094 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:00.040062 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a405563-5624-4dd8-ae28-cc610e6c64b8-maas-api-service-ca\") pod \"3a405563-5624-4dd8-ae28-cc610e6c64b8\" (UID: \"3a405563-5624-4dd8-ae28-cc610e6c64b8\") " Jul 29 08:11:00.040455 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:00.040429 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a405563-5624-4dd8-ae28-cc610e6c64b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a405563-5624-4dd8-ae28-cc610e6c64b8" (UID: "3a405563-5624-4dd8-ae28-cc610e6c64b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:11:00.141294 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:00.141195 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a405563-5624-4dd8-ae28-cc610e6c64b8-maas-api-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:00.805072 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:00.805019 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" Jul 29 08:11:00.805276 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:00.805019 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn" event={"ID":"3a405563-5624-4dd8-ae28-cc610e6c64b8","Type":"ContainerDied","Data":"5c6affc9f2644750dae1e56e4592a6a47dcba725b397c07ffc8e126e7c8bed77"} Jul 29 08:11:00.805276 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:00.805193 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6affc9f2644750dae1e56e4592a6a47dcba725b397c07ffc8e126e7c8bed77" Jul 29 08:11:01.945093 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:01.945029 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p"] Jul 29 08:11:01.945584 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:01.945309 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" podUID="ddf3fb74-101b-4524-90e7-cc05da325c5c" containerName="maas-api" containerID="cri-o://d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c" gracePeriod=30 Jul 29 08:11:01.981412 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:01.981383 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2"] Jul 29 08:11:01.989166 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:01.989137 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-71e038-29755200-mfkh2"] Jul 29 08:11:01.992847 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:01.992824 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:02.086558 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.086529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:02.197548 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.197487 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 08:11:02.251781 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.251744 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:02.259574 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.259543 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94hsf\" (UniqueName: \"kubernetes.io/projected/ddf3fb74-101b-4524-90e7-cc05da325c5c-kube-api-access-94hsf\") pod \"ddf3fb74-101b-4524-90e7-cc05da325c5c\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " Jul 29 08:11:02.259703 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.259581 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls\") pod \"ddf3fb74-101b-4524-90e7-cc05da325c5c\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " Jul 29 08:11:02.259779 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.259704 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf3fb74-101b-4524-90e7-cc05da325c5c-openshift-service-ca\") pod \"ddf3fb74-101b-4524-90e7-cc05da325c5c\" (UID: \"ddf3fb74-101b-4524-90e7-cc05da325c5c\") " Jul 29 08:11:02.260077 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.260033 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf3fb74-101b-4524-90e7-cc05da325c5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ddf3fb74-101b-4524-90e7-cc05da325c5c" (UID: "ddf3fb74-101b-4524-90e7-cc05da325c5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:11:02.261812 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.261765 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf3fb74-101b-4524-90e7-cc05da325c5c-kube-api-access-94hsf" (OuterVolumeSpecName: "kube-api-access-94hsf") pod "ddf3fb74-101b-4524-90e7-cc05da325c5c" (UID: "ddf3fb74-101b-4524-90e7-cc05da325c5c"). InnerVolumeSpecName "kube-api-access-94hsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:02.261910 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.261848 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ddf3fb74-101b-4524-90e7-cc05da325c5c" (UID: "ddf3fb74-101b-4524-90e7-cc05da325c5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:11:02.361278 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.361236 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf3fb74-101b-4524-90e7-cc05da325c5c-openshift-service-ca\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.361278 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.361274 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94hsf\" (UniqueName: \"kubernetes.io/projected/ddf3fb74-101b-4524-90e7-cc05da325c5c-kube-api-access-94hsf\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.361278 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.361286 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf3fb74-101b-4524-90e7-cc05da325c5c-maas-api-tls\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.383859 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.383822 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn"] Jul 29 08:11:02.388395 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.388368 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-71e038-b5whn"] Jul 29 08:11:02.389746 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.389717 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:02.738199 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.738165 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a405563-5624-4dd8-ae28-cc610e6c64b8" path="/var/lib/kubelet/pods/3a405563-5624-4dd8-ae28-cc610e6c64b8/volumes" Jul 29 08:11:02.738493 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.738478 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b35d97-ea13-4fb2-b5e4-c3737e8bce08" path="/var/lib/kubelet/pods/42b35d97-ea13-4fb2-b5e4-c3737e8bce08/volumes" Jul 29 08:11:02.802641 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.802611 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:02.816840 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.816804 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddf3fb74-101b-4524-90e7-cc05da325c5c" containerID="d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c" exitCode=0 Jul 29 08:11:02.817023 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.816869 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" Jul 29 08:11:02.817023 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.816877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" event={"ID":"ddf3fb74-101b-4524-90e7-cc05da325c5c","Type":"ContainerDied","Data":"d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c"} Jul 29 08:11:02.817023 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.816930 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p" event={"ID":"ddf3fb74-101b-4524-90e7-cc05da325c5c","Type":"ContainerDied","Data":"75f78ec62605870ca3869b87bd4ef0e566632816b66dae92744f1a4cddbfb4a1"} Jul 29 08:11:02.817023 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.816955 2568 scope.go:117] "RemoveContainer" containerID="d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c" Jul 29 08:11:02.828314 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.828289 2568 scope.go:117] "RemoveContainer" containerID="d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c" Jul 29 08:11:02.828638 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:11:02.828611 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c\": container with ID starting with d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c not found: ID does not exist" containerID="d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c" Jul 29 08:11:02.828714 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.828653 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c"} err="failed to get container status \"d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c\": rpc error: code = NotFound desc = could not find container \"d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c\": container with ID starting with d1a01e480ab8c22624617f42e06a2ca4dccf92e48b31d3dda32a023868e0429c not found: ID does not exist" Jul 29 08:11:02.838267 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.838238 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p"] Jul 29 08:11:02.842276 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:02.842250 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-71e038-6ddb457886-btw4p"] Jul 29 08:11:04.738488 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:04.738456 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf3fb74-101b-4524-90e7-cc05da325c5c" path="/var/lib/kubelet/pods/ddf3fb74-101b-4524-90e7-cc05da325c5c/volumes" Jul 29 08:11:09.827486 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:09.827445 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595"] Jul 29 08:11:09.827928 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:09.827822 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" containerID="cri-o://7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b" gracePeriod=30 Jul 29 08:11:10.084203 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:10.084115 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:10.183980 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:10.183948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:10.284645 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:10.284598 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:10.715172 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:10.715143 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84875955d7-l5s92_35d01ecf-5ddc-4d6d-ab32-db348d4540df/authorino/0.log" Jul 29 08:11:15.085530 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.085505 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 08:11:15.174362 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174328 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de31ccd-cb39-474d-914c-236f50281384-istio-podinfo\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174556 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174375 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174556 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174423 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-certs\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174556 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174467 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-data\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174556 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174505 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-socket\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174556 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174530 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-istio-token\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174819 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174584 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-envoy\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174987 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174962 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-credential-socket\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.174987 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174740 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:15.175217 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174825 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:15.175217 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.175010 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b28h\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-kube-api-access-4b28h\") pod \"9de31ccd-cb39-474d-914c-236f50281384\" (UID: \"9de31ccd-cb39-474d-914c-236f50281384\") " Jul 29 08:11:15.175377 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.175324 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-certs\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.175377 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.175346 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-workload-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.175377 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.174871 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:11:15.175377 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.175282 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:15.175579 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.175519 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-data" (OuterVolumeSpecName: "istio-data") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:15.176924 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.176898 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:15.177085 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.177064 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-kube-api-access-4b28h" (OuterVolumeSpecName: "kube-api-access-4b28h") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "kube-api-access-4b28h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:15.177207 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.177188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9de31ccd-cb39-474d-914c-236f50281384-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 08:11:15.177249 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.177204 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-istio-token" (OuterVolumeSpecName: "istio-token") pod "9de31ccd-cb39-474d-914c-236f50281384" (UID: "9de31ccd-cb39-474d-914c-236f50281384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:15.276008 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.275974 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-istio-token\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.276008 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.276003 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-envoy\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.276008 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.276012 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-credential-socket\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.276246 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.276021 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b28h\" (UniqueName: \"kubernetes.io/projected/9de31ccd-cb39-474d-914c-236f50281384-kube-api-access-4b28h\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.276246 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.276030 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de31ccd-cb39-474d-914c-236f50281384-istio-podinfo\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.276246 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.276063 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de31ccd-cb39-474d-914c-236f50281384-istiod-ca-cert\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.276246 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.276073 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de31ccd-cb39-474d-914c-236f50281384-istio-data\") on node \"ip-10-0-136-73.ec2.internal\" DevicePath \"\"" Jul 29 08:11:15.891867 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.891829 2568 generic.go:356] "Generic (PLEG): container finished" podID="9de31ccd-cb39-474d-914c-236f50281384" containerID="7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b" exitCode=0 Jul 29 08:11:15.892063 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.891883 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" Jul 29 08:11:15.892063 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.891913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" event={"ID":"9de31ccd-cb39-474d-914c-236f50281384","Type":"ContainerDied","Data":"7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b"} Jul 29 08:11:15.892063 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.891970 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595" event={"ID":"9de31ccd-cb39-474d-914c-236f50281384","Type":"ContainerDied","Data":"155639ae5f560a79e0811d014c88772f9677052c9b4980e79ee30f096abbb2f9"} Jul 29 08:11:15.892063 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.891993 2568 scope.go:117] "RemoveContainer" containerID="7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b" Jul 29 08:11:15.902286 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.902269 2568 scope.go:117] "RemoveContainer" containerID="6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939" Jul 29 08:11:15.910930 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.910911 2568 scope.go:117] "RemoveContainer" containerID="7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b" Jul 29 08:11:15.911226 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:11:15.911204 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b\": container with ID starting with 7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b not found: ID does not exist" containerID="7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b" Jul 29 08:11:15.911292 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.911233 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b"} err="failed to get container status \"7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b\": rpc error: code = NotFound desc = could not find container \"7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b\": container with ID starting with 7fd900a2401d20d29e65ea64a0287de350e8b4be18bc718731657a9d13bc287b not found: ID does not exist" Jul 29 08:11:15.911292 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.911251 2568 scope.go:117] "RemoveContainer" containerID="6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939" Jul 29 08:11:15.911500 ip-10-0-136-73 kubenswrapper[2568]: E0729 08:11:15.911486 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939\": container with ID starting with 6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939 not found: ID does not exist" containerID="6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939" Jul 29 08:11:15.911541 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.911504 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939"} err="failed to get container status \"6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939\": rpc error: code = NotFound desc = could not find container \"6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939\": container with ID starting with 6e5b2897a0eee3037ec9c6e84d9e117fe9ed60bb07c5e3b0209734e3a05f0939 not found: ID does not exist" Jul 29 08:11:15.923920 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.923898 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595"] Jul 29 08:11:15.931137 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:15.931105 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-71e038-openshift-default-5f4f5b57b4-zj595"] Jul 29 08:11:16.739548 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:16.739514 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de31ccd-cb39-474d-914c-236f50281384" path="/var/lib/kubelet/pods/9de31ccd-cb39-474d-914c-236f50281384/volumes" Jul 29 08:11:17.109574 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:17.109488 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cl2zp_9f6ba4b9-603b-430e-9d5c-023480140d23/manager/0.log" Jul 29 08:11:17.237532 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:17.237498 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-664896b984-x29n7_1c95489e-a155-46fb-923d-847be73a81dc/manager/0.log" Jul 29 08:11:17.357230 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:17.357192 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p2cn6_899184eb-665d-4185-b920-29233d02f5e3/server/0.log" Jul 29 08:11:17.477459 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:17.477425 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dcg5q_56b3b447-9444-4e22-ab2a-f38dc1244cb0/manager/0.log" Jul 29 08:11:17.762138 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:17.762066 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-g9gpc_b8b97b7a-9730-4448-8f5b-1823205df2b9/copy-manifests/0.log" Jul 29 08:11:17.769151 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:17.769127 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-g9gpc_b8b97b7a-9730-4448-8f5b-1823205df2b9/manager/0.log" Jul 29 08:11:18.294463 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:18.294436 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d46f59bcc-x5b6f_6c163dde-8e3c-43a1-8dc7-7a2176ad513d/maas-api/0.log" Jul 29 08:11:18.422148 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:18.422115 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755170-kd9fk_d022df94-7543-48ce-8101-60213328c804/cleanup/0.log" Jul 29 08:11:18.540070 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:18.540026 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755185-dj2cq_400f0ee4-df5f-4706-ab11-7b405a60d0a8/cleanup/0.log" Jul 29 08:11:18.664856 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:18.664827 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755200-jfwpx_d73ef970-2e12-446a-873d-0840a315813b/cleanup/0.log" Jul 29 08:11:18.791742 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:18.791717 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gcbw7_d9557216-c2c5-4946-80e0-fdc3c620c33e/postgres/0.log" Jul 29 08:11:19.694555 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:19.694525 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:20.151072 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:20.150958 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84875955d7-l5s92_35d01ecf-5ddc-4d6d-ab32-db348d4540df/authorino/0.log" Jul 29 08:11:20.279226 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:20.279200 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hdkjm_e954145a-c0c8-403f-b4f5-ffeafd0464c1/manager/0.log" Jul 29 08:11:20.413269 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:20.413239 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b5gtc_c286b451-056f-44b7-bf23-c5b26e3997f9/manager/0.log" Jul 29 08:11:20.671823 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:20.671741 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5mxs6_f709d68f-78af-49a9-9ee0-b5b628b42de1/registry-server/0.log" Jul 29 08:11:20.802190 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:20.802160 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-brgq9_89c236b5-5858-47f0-a849-51adee48c8f1/manager/0.log" Jul 29 08:11:21.043569 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:21.043492 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-n69t2_f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8/manager/0.log" Jul 29 08:11:21.667929 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:21.667901 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd888fd86-hl6vg_9e6a3c00-100f-4c4e-a5dd-1908529e2269/kube-auth-proxy/0.log" Jul 29 08:11:22.296094 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:22.296066 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6549448c57-9mrc9_1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7/router/0.log" Jul 29 08:11:22.805567 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:22.805538 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-s9c4p_68de6b1a-ab33-41a6-86bf-fbb7c0ec59dc/main/0.log" Jul 29 08:11:28.705377 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705335 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ztqbj/must-gather-xvb6b"] Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705722 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705733 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705748 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf3fb74-101b-4524-90e7-cc05da325c5c" containerName="maas-api" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705753 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf3fb74-101b-4524-90e7-cc05da325c5c" containerName="maas-api" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705767 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705773 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88206e4-4cb8-43ac-a2dd-5012efa6c85d" containerName="istio-proxy" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705779 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a405563-5624-4dd8-ae28-cc610e6c64b8" containerName="revoke-keys" Jul 29 08:11:28.705788 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705784 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a405563-5624-4dd8-ae28-cc610e6c64b8" containerName="revoke-keys" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705850 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705857 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b836a79e-dc47-4f1a-8461-be8412776e3b" containerName="istio-proxy" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705866 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a405563-5624-4dd8-ae28-cc610e6c64b8" containerName="revoke-keys" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705871 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf3fb74-101b-4524-90e7-cc05da325c5c" containerName="maas-api" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705943 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.705949 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" Jul 29 08:11:28.706122 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.706016 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de31ccd-cb39-474d-914c-236f50281384" containerName="istio-proxy" Jul 29 08:11:28.710830 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.710810 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:28.715607 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.715586 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ztqbj\"/\"default-dockercfg-jwhlp\"" Jul 29 08:11:28.715719 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.715586 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ztqbj\"/\"kube-root-ca.crt\"" Jul 29 08:11:28.716353 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.716336 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ztqbj\"/\"openshift-service-ca.crt\"" Jul 29 08:11:28.724464 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.724434 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztqbj/must-gather-xvb6b"] Jul 29 08:11:28.724920 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.724893 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:28.793194 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.793153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/840f26ef-d06b-49d0-bd7a-64a57f3db837-must-gather-output\") pod \"must-gather-xvb6b\" (UID: \"840f26ef-d06b-49d0-bd7a-64a57f3db837\") " pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:28.793194 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.793198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtv7z\" (UniqueName: \"kubernetes.io/projected/840f26ef-d06b-49d0-bd7a-64a57f3db837-kube-api-access-xtv7z\") pod \"must-gather-xvb6b\" (UID: \"840f26ef-d06b-49d0-bd7a-64a57f3db837\") " pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:28.819916 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.819891 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:28.894447 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.894412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/840f26ef-d06b-49d0-bd7a-64a57f3db837-must-gather-output\") pod \"must-gather-xvb6b\" (UID: \"840f26ef-d06b-49d0-bd7a-64a57f3db837\") " pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:28.894447 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.894445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtv7z\" (UniqueName: \"kubernetes.io/projected/840f26ef-d06b-49d0-bd7a-64a57f3db837-kube-api-access-xtv7z\") pod \"must-gather-xvb6b\" (UID: \"840f26ef-d06b-49d0-bd7a-64a57f3db837\") " pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:28.894784 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.894762 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/840f26ef-d06b-49d0-bd7a-64a57f3db837-must-gather-output\") pod \"must-gather-xvb6b\" (UID: \"840f26ef-d06b-49d0-bd7a-64a57f3db837\") " pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:28.903766 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:28.903742 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtv7z\" (UniqueName: \"kubernetes.io/projected/840f26ef-d06b-49d0-bd7a-64a57f3db837-kube-api-access-xtv7z\") pod \"must-gather-xvb6b\" (UID: \"840f26ef-d06b-49d0-bd7a-64a57f3db837\") " pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:29.020850 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:29.020758 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztqbj/must-gather-xvb6b" Jul 29 08:11:29.187350 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:29.187320 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztqbj/must-gather-xvb6b"] Jul 29 08:11:29.189397 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:11:29.189371 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod840f26ef_d06b_49d0_bd7a_64a57f3db837.slice/crio-e523381060648ab40aec38d8ea5fceebfd23db47096c7c5e0caaa2ed4967b5d6 WatchSource:0}: Error finding container e523381060648ab40aec38d8ea5fceebfd23db47096c7c5e0caaa2ed4967b5d6: Status 404 returned error can't find the container with id e523381060648ab40aec38d8ea5fceebfd23db47096c7c5e0caaa2ed4967b5d6 Jul 29 08:11:29.191162 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:29.191143 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:11:29.325238 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:29.325141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" limit=1024 Jul 29 08:11:29.965348 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:29.965312 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/must-gather-xvb6b" event={"ID":"840f26ef-d06b-49d0-bd7a-64a57f3db837","Type":"ContainerStarted","Data":"e523381060648ab40aec38d8ea5fceebfd23db47096c7c5e0caaa2ed4967b5d6"} Jul 29 08:11:30.974897 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:30.974847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/must-gather-xvb6b" event={"ID":"840f26ef-d06b-49d0-bd7a-64a57f3db837","Type":"ContainerStarted","Data":"c172e398c791c318da0f9acda82c7426ba44ea60e1dc9e70c2139ccbba53c8f8"} Jul 29 08:11:30.975533 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:30.975500 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/must-gather-xvb6b" event={"ID":"840f26ef-d06b-49d0-bd7a-64a57f3db837","Type":"ContainerStarted","Data":"59f6102a651c069c7454604f3b61e36a564477d8be8290df4e32fae4d1b96276"} Jul 29 08:11:30.994260 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:30.994207 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ztqbj/must-gather-xvb6b" podStartSLOduration=1.764460041 podStartE2EDuration="2.994191422s" totalImagesPullingTime="1.229731381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:11:30.992526161 +0000 UTC m=+3400.763976973" watchObservedRunningTime="2026-07-29 08:11:30.994191422 +0000 UTC m=+3400.765642221" Jul 29 08:11:32.005852 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:32.005811 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-82z89_c71bd361-1cda-4ccd-8daf-3fc786c6d773/global-pull-secret-syncer/0.log" Jul 29 08:11:32.169478 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:32.169437 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6wm6z_40135fd0-f3bb-40d5-986b-9530c70706fa/konnectivity-agent/0.log" Jul 29 08:11:32.312269 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:32.312185 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-73.ec2.internal_1111a70e9f15469ee95cf515ced571c7/haproxy/0.log" Jul 29 08:11:38.006794 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:38.006384 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84875955d7-l5s92_35d01ecf-5ddc-4d6d-ab32-db348d4540df/authorino/0.log" Jul 29 08:11:38.054271 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:38.054238 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hdkjm_e954145a-c0c8-403f-b4f5-ffeafd0464c1/manager/0.log" Jul 29 08:11:38.124840 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:38.124804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b5gtc_c286b451-056f-44b7-bf23-c5b26e3997f9/manager/0.log" Jul 29 08:11:38.252923 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:38.252885 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5mxs6_f709d68f-78af-49a9-9ee0-b5b628b42de1/registry-server/0.log" Jul 29 08:11:38.504482 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:38.504448 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-brgq9_89c236b5-5858-47f0-a849-51adee48c8f1/manager/0.log" Jul 29 08:11:38.705419 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:38.705381 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-n69t2_f9d393ce-18fe-4870-9b95-2a2e8bdaf8c8/manager/0.log" Jul 29 08:11:40.566991 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:40.566955 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-lqljb_1ac2baf2-1488-4dc6-baec-f6f313f330e0/cluster-monitoring-operator/0.log" Jul 29 08:11:40.839641 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:40.839555 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw92m_80278e74-48be-4578-84fd-0956760475b2/node-exporter/0.log" Jul 29 08:11:40.861912 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:40.861885 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw92m_80278e74-48be-4578-84fd-0956760475b2/kube-rbac-proxy/0.log" Jul 29 08:11:40.883758 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:40.883734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw92m_80278e74-48be-4578-84fd-0956760475b2/init-textfile/0.log" Jul 29 08:11:42.602186 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:42.602105 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-sqbgr_7154b5ee-985a-4b27-96e7-34cd469b9b69/networking-console-plugin/0.log" Jul 29 08:11:43.222958 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.222918 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj"] Jul 29 08:11:43.229724 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.229689 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.265057 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.265000 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj"] Jul 29 08:11:43.267654 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.267620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-proc\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.267820 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.267740 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-lib-modules\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.267820 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.267767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-podres\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.267820 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.267803 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-sys\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.268011 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.267865 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88ct\" (UniqueName: \"kubernetes.io/projected/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-kube-api-access-h88ct\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.306720 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.306689 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/1.log" Jul 29 08:11:43.312777 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.312750 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ttnwj_7a9c8dc2-3c50-48e2-a097-7313bfbd4de3/console-operator/2.log" Jul 29 08:11:43.368692 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-lib-modules\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368692 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-podres\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-sys\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88ct\" (UniqueName: \"kubernetes.io/projected/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-kube-api-access-h88ct\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-proc\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368853 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-lib-modules\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368926 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-proc\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.368963 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.368928 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-sys\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.369281 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.369010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-podres\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.380616 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.380558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88ct\" (UniqueName: \"kubernetes.io/projected/3bab88c7-ed0a-4805-9dc2-561b9d0ed41a-kube-api-access-h88ct\") pod \"perf-node-gather-daemonset-krvwj\" (UID: \"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.540251 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.540175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:43.726443 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.726413 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj"] Jul 29 08:11:43.727673 ip-10-0-136-73 kubenswrapper[2568]: W0729 08:11:43.727642 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3bab88c7_ed0a_4805_9dc2_561b9d0ed41a.slice/crio-da953478aff47ff0694c099e6909aaf30d6d0ab46950d57c1442245062b083a5 WatchSource:0}: Error finding container da953478aff47ff0694c099e6909aaf30d6d0ab46950d57c1442245062b083a5: Status 404 returned error can't find the container with id da953478aff47ff0694c099e6909aaf30d6d0ab46950d57c1442245062b083a5 Jul 29 08:11:43.972344 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:43.972312 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-ldld4_e31b612f-6955-4087-a735-b5e82e5dbb83/download-server/0.log" Jul 29 08:11:44.086153 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:44.086096 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" event={"ID":"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a","Type":"ContainerStarted","Data":"f6054cf3a96ad4110e6053d1e45be7ec52eda46fc9c1deb7ac5df95c0ca87367"} Jul 29 08:11:44.086372 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:44.086190 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" event={"ID":"3bab88c7-ed0a-4805-9dc2-561b9d0ed41a","Type":"ContainerStarted","Data":"da953478aff47ff0694c099e6909aaf30d6d0ab46950d57c1442245062b083a5"} Jul 29 08:11:44.086372 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:44.086222 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:44.110288 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:44.110229 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" podStartSLOduration=1.11021021 podStartE2EDuration="1.11021021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:11:44.10707362 +0000 UTC m=+3413.878524411" watchObservedRunningTime="2026-07-29 08:11:44.11021021 +0000 UTC m=+3413.881661009" Jul 29 08:11:44.717186 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:44.717156 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-8qjrq_5f8893ab-e2aa-424e-9f9d-a23051175228/volume-data-source-validator/0.log" Jul 29 08:11:46.203119 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:46.203085 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkkwv_2eac22f6-6f8c-499c-ac0f-efb9459915e6/dns/0.log" Jul 29 08:11:46.240403 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:46.240374 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkkwv_2eac22f6-6f8c-499c-ac0f-efb9459915e6/kube-rbac-proxy/0.log" Jul 29 08:11:46.339631 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:46.339604 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z22p2_35d5b11f-9aca-4a3d-a010-c9cde5be22d4/dns-node-resolver/0.log" Jul 29 08:11:47.119035 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:47.119006 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8zxgt_9045afd8-91a2-4b61-8424-19c537bd41ee/node-ca/0.log" Jul 29 08:11:48.683861 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:48.683824 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd888fd86-hl6vg_9e6a3c00-100f-4c4e-a5dd-1908529e2269/kube-auth-proxy/0.log" Jul 29 08:11:48.971587 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:48.971513 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6549448c57-9mrc9_1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7/router/0.log" Jul 29 08:11:49.967888 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:49.967857 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qrcg8_5edede8b-3ffc-4a62-873a-8efa06389841/serve-healthcheck-canary/0.log" Jul 29 08:11:50.104068 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:50.104024 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-krvwj" Jul 29 08:11:50.547736 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:50.547701 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-gwn5p_2abf6566-aa10-4aec-bf74-26f935f88cda/insights-operator/0.log" Jul 29 08:11:50.550357 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:50.550328 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-gwn5p_2abf6566-aa10-4aec-bf74-26f935f88cda/insights-operator/1.log" Jul 29 08:11:50.746376 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:50.746327 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7fzh_2613fe87-dc24-45d8-8897-f211058dbd51/kube-rbac-proxy/0.log" Jul 29 08:11:50.771335 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:50.771311 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7fzh_2613fe87-dc24-45d8-8897-f211058dbd51/exporter/0.log" Jul 29 08:11:50.810591 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:50.810473 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7fzh_2613fe87-dc24-45d8-8897-f211058dbd51/extractor/0.log" Jul 29 08:11:51.403635 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:51.403605 2568 scope.go:117] "RemoveContainer" containerID="e2fb140dd3a06eae1ada07f47f23199f73302841ba503b52b05be5cdceca90e9" Jul 29 08:11:52.831824 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:52.831772 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd888fd86-hl6vg_9e6a3c00-100f-4c4e-a5dd-1908529e2269/kube-auth-proxy/0.log" Jul 29 08:11:52.981452 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:52.981415 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6549448c57-9mrc9_1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7/router/0.log" Jul 29 08:11:53.736701 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:53.736666 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cl2zp_9f6ba4b9-603b-430e-9d5c-023480140d23/manager/0.log" Jul 29 08:11:54.010377 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.010336 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-664896b984-x29n7_1c95489e-a155-46fb-923d-847be73a81dc/manager/0.log" Jul 29 08:11:54.033793 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.033767 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p2cn6_899184eb-665d-4185-b920-29233d02f5e3/server/0.log" Jul 29 08:11:54.132229 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.132196 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dcg5q_56b3b447-9444-4e22-ab2a-f38dc1244cb0/manager/0.log" Jul 29 08:11:54.365734 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.365651 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-g9gpc_b8b97b7a-9730-4448-8f5b-1823205df2b9/manager/0.log" Jul 29 08:11:54.422921 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.422894 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-g9gpc_b8b97b7a-9730-4448-8f5b-1823205df2b9/copy-manifests/0.log" Jul 29 08:11:54.719451 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.719422 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd888fd86-hl6vg_9e6a3c00-100f-4c4e-a5dd-1908529e2269/kube-auth-proxy/0.log" Jul 29 08:11:54.990983 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:11:54.990896 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6549448c57-9mrc9_1cfb2fcd-e0a9-4feb-b596-3849b1e2e8b7/router/0.log" Jul 29 08:12:00.698119 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:12:00.698088 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-69wtg_e91e5b20-9602-4675-8adb-0665c161de9a/migrator/0.log" Jul 29 08:12:00.718659 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:12:00.718629 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-69wtg_e91e5b20-9602-4675-8adb-0665c161de9a/graceful-termination/0.log" Jul 29 08:12:01.149815 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:12:01.149727 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp_a2ee561b-df1e-47b7-a495-c6fee6888ec5/kube-storage-version-migrator-operator/1.log" Jul 29 08:12:01.150466 ip-10-0-136-73 kubenswrapper[2568]: I0729 08:12:01.150439 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-6p8wp_a2ee561b-df1e-47b7-a495-c6fee6888ec5/kube-storage-version-migrator-operator/0.log"