Jul 10 15:55:09.811090 ip-10-0-139-69 systemd[1]: Starting Kubernetes Kubelet... Jul 10 15:55:10.220411 ip-10-0-139-69 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 15:55:10.220411 ip-10-0-139-69 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 15:55:10.220411 ip-10-0-139-69 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 15:55:10.220411 ip-10-0-139-69 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 15:55:10.220411 ip-10-0-139-69 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 15:55:10.223398 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.223222 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 15:55:10.226937 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226917 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 15:55:10.226937 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226936 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 15:55:10.226937 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226941 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226945 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226949 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226951 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226954 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226957 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226960 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226962 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226965 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226968 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226970 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226973 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226976 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226978 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226982 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226985 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226995 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.226997 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227000 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227002 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 15:55:10.227072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227005 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227008 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227010 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227012 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227015 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227017 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227020 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227022 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227025 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227027 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227030 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227033 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227036 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227041 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227045 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227048 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227051 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227054 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227057 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 15:55:10.227572 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227060 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227063 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227066 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227069 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227071 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227074 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227077 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227080 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227083 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227086 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227088 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227091 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227093 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227096 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227098 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227101 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227103 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227106 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227108 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227111 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 15:55:10.228033 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227113 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227116 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227118 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227121 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227123 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227126 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227128 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227132 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227135 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227138 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227141 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227143 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227146 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227148 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227151 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227154 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227156 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227159 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227162 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227165 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 15:55:10.228528 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227167 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227170 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227173 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227175 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227179 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227182 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227184 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227187 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227189 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227192 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227194 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227196 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227199 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227201 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227206 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227209 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227212 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227215 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227958 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 15:55:10.229015 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227966 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227969 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227972 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227975 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227977 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227981 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227984 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227986 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227989 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227991 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227994 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227996 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.227999 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228004 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228008 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228011 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228014 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228017 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228019 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 15:55:10.229468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228023 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228027 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228030 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228033 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228035 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228038 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228041 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228043 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228046 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228048 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228051 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228053 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228056 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228059 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228062 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228065 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228067 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228070 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228072 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228075 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 15:55:10.229983 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228077 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228080 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228082 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228085 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228087 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228089 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228092 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228095 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228098 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228100 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228103 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228105 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228107 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228110 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228113 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228115 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228118 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228120 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228123 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 15:55:10.230468 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228126 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228129 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228132 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228134 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228137 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228141 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228143 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228146 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228148 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228151 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228154 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228156 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228159 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228161 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228164 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228166 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228169 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228172 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228174 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228177 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 15:55:10.230948 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228180 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228183 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228186 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228189 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228191 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228193 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228196 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228198 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228200 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228204 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228206 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228209 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228213 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228216 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228219 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228223 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228225 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228228 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228236 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228239 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 15:55:10.231435 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228708 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228715 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228718 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228721 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228725 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228729 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228732 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228735 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228738 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228740 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228743 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228746 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228748 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228752 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228754 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228757 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228760 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228762 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228765 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228767 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 15:55:10.231933 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228770 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228788 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228792 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228796 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228799 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228802 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228805 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228807 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228810 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228813 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228818 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228821 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228825 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228829 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228831 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228834 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228837 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228840 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228843 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 15:55:10.232414 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228846 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228849 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228851 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228854 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228857 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228859 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228862 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228870 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228873 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228876 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228878 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228881 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228889 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228891 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228894 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228897 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228899 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228902 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228904 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 15:55:10.232892 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228907 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228909 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228911 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228915 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228919 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228923 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228927 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228930 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228932 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228936 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228939 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228943 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228946 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228948 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228951 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228954 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228957 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228959 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228962 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 15:55:10.233347 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228965 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228967 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228971 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228973 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228976 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228978 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228981 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228983 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228986 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228988 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228990 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228993 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228995 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.228998 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229000 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229002 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229005 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229009 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229012 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229015 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 15:55:10.233816 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229017 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229020 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229099 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229107 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229115 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229122 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229129 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229134 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229141 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229146 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229149 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229152 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229156 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229159 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229162 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229166 2576 flags.go:64] FLAG: --cgroup-root="" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229169 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229172 2576 flags.go:64] FLAG: --client-ca-file="" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229175 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229178 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229182 2576 flags.go:64] FLAG: --cluster-domain="" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229185 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229188 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229191 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 15:55:10.234293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229195 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229199 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229201 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229205 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229208 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229212 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229215 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229217 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229221 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229224 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229228 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229231 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229234 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229237 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229240 2576 flags.go:64] FLAG: --enable-server="true" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229243 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229247 2576 flags.go:64] FLAG: --event-burst="100" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229250 2576 flags.go:64] FLAG: --event-qps="50" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229253 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229255 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229258 2576 flags.go:64] FLAG: --eviction-hard="" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229263 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229265 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229268 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229272 2576 flags.go:64] FLAG: --eviction-soft="" Jul 10 15:55:10.234872 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229275 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229277 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229280 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229285 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229287 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229290 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229293 2576 flags.go:64] FLAG: --feature-gates="" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229297 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229300 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229303 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229306 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229309 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229312 2576 flags.go:64] FLAG: --help="false" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229315 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229318 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229321 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229324 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229328 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229331 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229334 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229337 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229340 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229342 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 15:55:10.235495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229345 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229349 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229351 2576 flags.go:64] FLAG: --kube-reserved="" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229354 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229357 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229360 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229363 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229365 2576 flags.go:64] FLAG: --lock-file="" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229368 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229372 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229375 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229380 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229385 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229387 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229390 2576 flags.go:64] FLAG: --logging-format="text" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229393 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229396 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229399 2576 flags.go:64] FLAG: --manifest-url="" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229402 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229407 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229409 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229414 2576 flags.go:64] FLAG: --max-pods="110" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229417 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229420 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229423 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 15:55:10.236188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229426 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229429 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229432 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229435 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229442 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229445 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229448 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229451 2576 flags.go:64] FLAG: --pod-cidr="" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229454 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229460 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229463 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229466 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229469 2576 flags.go:64] FLAG: --port="10250" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229472 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229475 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ad718c1e2b0d921" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229479 2576 flags.go:64] FLAG: --qos-reserved="" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229482 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229485 2576 flags.go:64] FLAG: --register-node="true" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229488 2576 flags.go:64] FLAG: --register-with-taints="" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229494 2576 flags.go:64] FLAG: --registry-burst="10" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229496 2576 flags.go:64] FLAG: --registry-qps="5" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229499 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229502 2576 flags.go:64] FLAG: --reserved-memory="" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229507 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229510 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 15:55:10.236854 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229512 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229515 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229518 2576 flags.go:64] FLAG: --runonce="false" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229520 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229523 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229526 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229529 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229532 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229534 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229538 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229541 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229543 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229546 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229549 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229552 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229554 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229557 2576 flags.go:64] FLAG: --system-cgroups="" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229560 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229565 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229568 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229571 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229577 2576 flags.go:64] FLAG: --tls-min-version="" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229580 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229583 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229585 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 15:55:10.237463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229589 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229594 2576 flags.go:64] FLAG: --v="2" Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229598 2576 flags.go:64] FLAG: --version="false" Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229602 2576 flags.go:64] FLAG: --vmodule="" Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229606 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.229609 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229730 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229735 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229738 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229741 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229744 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229746 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229749 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229752 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229754 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229757 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229759 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229762 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229764 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229767 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229769 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 15:55:10.238072 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229785 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229788 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229793 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229796 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229799 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229802 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229806 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229811 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229813 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229816 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229819 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229821 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229826 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229828 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229831 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229835 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229838 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229841 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 15:55:10.238612 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229844 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229847 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229849 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229852 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229855 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229857 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229860 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229863 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229865 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229868 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229870 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229873 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229876 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229878 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229880 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229883 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229886 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229888 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229891 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229894 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 15:55:10.239060 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229896 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229900 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229903 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229906 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229908 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229911 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229915 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229918 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229920 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229923 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229925 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229928 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229931 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229933 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229936 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229939 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229941 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229944 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229946 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229949 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 15:55:10.239531 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229951 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229953 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229956 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229958 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229961 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229963 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229966 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229968 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229971 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229973 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229975 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229978 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229980 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229984 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229987 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229990 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229992 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229995 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.229999 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230002 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230005 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 15:55:10.240020 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230007 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230010 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230012 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230015 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.230017 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.230689 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.238102 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.238124 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.238143 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.238148 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238210 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238216 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238219 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238222 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238224 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 15:55:10.240538 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238227 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238230 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238233 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238236 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238238 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238241 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238244 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238246 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238249 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238253 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238257 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238260 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238263 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238265 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238268 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238270 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238272 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238275 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238277 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 15:55:10.240915 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238280 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238283 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238285 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238288 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238293 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238298 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238302 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238305 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238308 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238311 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238314 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238317 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238319 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238322 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238325 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238327 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238330 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238332 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 15:55:10.241432 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238335 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238338 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238340 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238343 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238346 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238348 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238351 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238354 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238356 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238359 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238362 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238365 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238368 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238370 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238373 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238375 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238378 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238381 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238383 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238386 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 15:55:10.241889 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238389 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238392 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238394 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238397 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238399 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238402 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238404 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238407 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238409 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238412 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238414 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238416 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238419 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238421 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238424 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238426 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238428 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238431 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238433 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238437 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238439 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 15:55:10.242359 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238441 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238444 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238447 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238449 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238452 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238454 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238457 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238459 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238462 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238465 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238467 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238470 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238473 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238476 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238480 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 15:55:10.242968 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238483 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.238488 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238606 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238612 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238615 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238618 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238621 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238624 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238626 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238630 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238632 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238635 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238638 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238640 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238643 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 15:55:10.243318 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238645 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238648 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238651 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238653 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238656 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238658 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238661 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238663 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238666 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238668 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238671 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238673 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238677 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238679 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238682 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238685 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238687 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238690 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238692 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238695 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 15:55:10.243686 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238697 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238700 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238702 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238706 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238710 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238713 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238716 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238719 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238722 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238725 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238728 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238730 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238733 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238736 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238751 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238754 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238757 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238759 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238762 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 15:55:10.244261 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238766 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238769 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238772 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238787 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238790 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238793 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238795 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238798 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238802 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238806 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238810 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238813 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238815 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238818 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238820 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238823 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238825 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238828 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238830 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238833 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 15:55:10.244878 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238836 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238838 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238841 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238843 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238845 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238848 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238851 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238853 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238856 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238858 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238861 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238863 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238865 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238868 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238870 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238873 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238875 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238878 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238881 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238883 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 15:55:10.245696 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238886 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238888 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238892 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238895 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238897 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238900 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:10.238902 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.238907 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.239055 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.241143 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.242445 2576 server.go:1019] "Starting client certificate rotation" Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.242567 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.242613 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 15:55:10.246277 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.245091 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 15:55:10.247908 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.247886 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 15:55:10.261149 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.261123 2576 log.go:25] "Validated CRI v1 runtime API" Jul 10 15:55:10.263033 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.263018 2576 log.go:25] "Validated CRI v1 image API" Jul 10 15:55:10.264524 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.264506 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 15:55:10.269599 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.269577 2576 fs.go:135] Filesystem UUIDs: map[0379ff5c-3671-4940-bf95-10255cf95577:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ca8f6cb2-02c9-4980-b3b5-04add7be61b1:/dev/nvme0n1p4] Jul 10 15:55:10.269681 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.269597 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 15:55:10.275513 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.275399 2576 manager.go:216] Machine: {Timestamp:2026-07-10 15:55:10.273318224 +0000 UTC m=+0.345611443 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079080 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec203e36c2194c8b218207016123b0f3 SystemUUID:ec203e36-c219-4c8b-2182-07016123b0f3 BootID:b135fa4b-32d3-4323-8030-69545d25b4c6 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:25:41:e2:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:25:41:e2:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:49:e0:32:c7:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 15:55:10.275513 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.275508 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 15:55:10.275656 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.275644 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 15:55:10.277493 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.277476 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 15:55:10.278226 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.278189 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 15:55:10.278365 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.278230 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-69.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 15:55:10.278909 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.278899 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 15:55:10.278947 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.278912 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 15:55:10.278947 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.278924 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 15:55:10.280228 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.280216 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 15:55:10.281376 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.281365 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 15:55:10.284092 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284080 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 10 15:55:10.284391 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284381 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 15:55:10.284428 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284413 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 10 15:55:10.284428 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284424 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 15:55:10.284493 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284434 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 15:55:10.284493 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284454 2576 kubelet.go:398] "Adding apiserver pod source" Jul 10 15:55:10.284493 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284465 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 15:55:10.285028 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.284861 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 15:55:10.285109 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.285040 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 15:55:10.288400 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.288385 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 15:55:10.290348 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290335 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 15:55:10.290396 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290366 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 15:55:10.290513 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290503 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 15:55:10.290542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290518 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 15:55:10.290542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290531 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 15:55:10.290542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290537 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 15:55:10.290619 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290545 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 15:55:10.290619 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290550 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 15:55:10.290964 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290955 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 15:55:10.290964 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290964 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 15:55:10.291019 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290971 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 15:55:10.291019 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290976 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 15:55:10.291019 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.290991 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 15:55:10.291019 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.291001 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 15:55:10.291713 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.291704 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 15:55:10.291713 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.291713 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 15:55:10.295066 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.295051 2576 server.go:1268] "Started kubelet" Jul 10 15:55:10.295222 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.295136 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 15:55:10.295289 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.295242 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 15:55:10.295331 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.295135 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 15:55:10.295907 ip-10-0-139-69 systemd[1]: Started Kubernetes Kubelet. Jul 10 15:55:10.296660 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.296641 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 15:55:10.297307 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.297293 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 10 15:55:10.300836 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.300725 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-69.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 15:55:10.301233 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.301208 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 15:55:10.301502 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.301474 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 15:55:10.302539 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.302513 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 15:55:10.304504 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.304484 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 15:55:10.304504 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.303532 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-69.ec2.internal.18c0f8b55e62dfdc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-69.ec2.internal,UID:ip-10-0-139-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-69.ec2.internal,},FirstTimestamp:2026-07-10 15:55:10.29502358 +0000 UTC m=+0.367316799,LastTimestamp:2026-07-10 15:55:10.29502358 +0000 UTC m=+0.367316799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-69.ec2.internal,}" Jul 10 15:55:10.304658 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.304491 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 15:55:10.304658 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.304605 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.305295 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305273 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 15:55:10.305295 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305294 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 15:55:10.305429 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305371 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 10 15:55:10.305429 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305382 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 10 15:55:10.305429 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305396 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 15:55:10.305559 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305518 2576 factory.go:55] Registering systemd factory Jul 10 15:55:10.305559 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305534 2576 factory.go:223] Registration of the systemd container factory successfully Jul 10 15:55:10.305793 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305744 2576 factory.go:153] Registering CRI-O factory Jul 10 15:55:10.305877 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305822 2576 factory.go:223] Registration of the crio container factory successfully Jul 10 15:55:10.305999 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.305988 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 15:55:10.306044 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.306014 2576 factory.go:103] Registering Raw factory Jul 10 15:55:10.306044 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.306028 2576 manager.go:1196] Started watching for new ooms in manager Jul 10 15:55:10.306910 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.306894 2576 manager.go:349] Starting recovery of all containers Jul 10 15:55:10.313921 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.313892 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-blgb8" Jul 10 15:55:10.314073 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.314049 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 15:55:10.314757 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.314702 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 15:55:10.317434 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.317411 2576 manager.go:354] Recovery completed Jul 10 15:55:10.318556 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.318537 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-blgb8" Jul 10 15:55:10.322445 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.322432 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.324519 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.324505 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.324587 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.324533 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.324587 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.324544 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.325021 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325007 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 15:55:10.325021 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325020 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.325130 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325020 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 15:55:10.325130 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325085 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 10 15:55:10.325799 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325757 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.325871 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325824 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.325871 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.325838 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.326093 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.326076 2576 policy_none.go:49] "None policy: Start" Jul 10 15:55:10.326093 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.326094 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 15:55:10.326196 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.326104 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 15:55:10.326983 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.326970 2576 policy_none.go:47] "Start" Jul 10 15:55:10.327241 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.327167 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-69.ec2.internal.18c0f8b56024f45c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-69.ec2.internal,UID:ip-10-0-139-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-69.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-69.ec2.internal,},FirstTimestamp:2026-07-10 15:55:10.324520028 +0000 UTC m=+0.396813247,LastTimestamp:2026-07-10 15:55:10.324520028 +0000 UTC m=+0.396813247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-69.ec2.internal,}" Jul 10 15:55:10.372458 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.372425 2576 manager.go:337] "Starting Device Plugin manager" Jul 10 15:55:10.372596 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.372548 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 15:55:10.372596 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.372565 2576 server.go:87] "Starting device plugin registration server" Jul 10 15:55:10.372901 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.372888 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 15:55:10.372962 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.372907 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 15:55:10.373021 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.372997 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 15:55:10.373121 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.373103 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 15:55:10.373121 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.373122 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 15:55:10.373767 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.373744 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 15:55:10.373854 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.373811 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.473081 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.472993 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.474283 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.474016 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.474283 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.474044 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.474283 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.474054 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.474283 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.474080 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.482899 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.482871 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.483040 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.482904 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-69.ec2.internal\": node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.485767 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.485728 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 15:55:10.487194 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.487178 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 15:55:10.487273 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.487216 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 15:55:10.487273 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.487239 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 15:55:10.487363 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.487276 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 15:55:10.490024 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.490007 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 15:55:10.521232 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.521208 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.587403 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.587347 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal"] Jul 10 15:55:10.587498 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.587455 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.594662 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.594645 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.594737 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.594683 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.594737 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.594696 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.596034 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596021 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.596142 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.596193 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596158 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.596867 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596849 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.596959 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596879 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.596959 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596890 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.596959 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596851 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.596959 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596945 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.596959 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.596956 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.598091 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.598074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.598183 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.598128 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 15:55:10.598910 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.598896 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 15:55:10.598992 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.598926 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 15:55:10.598992 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.598941 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 15:55:10.607319 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.607302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af23fa6bfd969eb2ea8a7d64ea7106f7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-69.ec2.internal\" (UID: \"af23fa6bfd969eb2ea8a7d64ea7106f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.607397 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.607326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6479ce33261f7d671487597879e80661-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal\" (UID: \"6479ce33261f7d671487597879e80661\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.607397 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.607342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6479ce33261f7d671487597879e80661-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal\" (UID: \"6479ce33261f7d671487597879e80661\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.616454 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.616426 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-69.ec2.internal\" not found" node="ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.620092 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.620070 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-69.ec2.internal\" not found" node="ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.622112 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.622096 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.708428 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.708391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6479ce33261f7d671487597879e80661-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal\" (UID: \"6479ce33261f7d671487597879e80661\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.708428 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.708424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6479ce33261f7d671487597879e80661-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal\" (UID: \"6479ce33261f7d671487597879e80661\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.708634 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.708441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af23fa6bfd969eb2ea8a7d64ea7106f7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-69.ec2.internal\" (UID: \"af23fa6bfd969eb2ea8a7d64ea7106f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.708634 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.708497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6479ce33261f7d671487597879e80661-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal\" (UID: \"6479ce33261f7d671487597879e80661\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.708634 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.708503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6479ce33261f7d671487597879e80661-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal\" (UID: \"6479ce33261f7d671487597879e80661\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.708634 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.708554 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af23fa6bfd969eb2ea8a7d64ea7106f7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-69.ec2.internal\" (UID: \"af23fa6bfd969eb2ea8a7d64ea7106f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.723237 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.723180 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.823821 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.823768 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:10.918383 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.918347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.923381 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:10.923365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" Jul 10 15:55:10.924462 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:10.924432 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:12.026077 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.026041 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:12.126621 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.126578 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-69.ec2.internal\" not found" Jul 10 15:55:12.151232 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.151202 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 15:55:12.206531 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.206490 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" Jul 10 15:55:12.220141 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.220116 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 15:55:12.221908 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.221890 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" Jul 10 15:55:12.231477 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.231452 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 15:55:12.243668 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.243630 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 15:55:12.243860 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.243831 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 15:55:12.265802 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.265725 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 15:55:12.286352 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.286326 2576 apiserver.go:52] "Watching apiserver" Jul 10 15:55:12.292635 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.292612 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 15:55:12.294498 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.294477 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv","openshift-cluster-node-tuning-operator/tuned-c8pxv","openshift-dns/node-resolver-bwg7j","openshift-image-registry/node-ca-28vk6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal","openshift-multus/multus-additional-cni-plugins-hkh5t","openshift-multus/multus-hrjh6","openshift-network-diagnostics/network-check-target-ddv99","kube-system/global-pull-secret-syncer-kw467","kube-system/konnectivity-agent-g9mg9","kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal","openshift-multus/network-metrics-daemon-85zw9","openshift-network-operator/iptables-alerter-pgtjw","openshift-ovn-kubernetes/ovnkube-node-ssxhb"] Jul 10 15:55:12.295819 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.295798 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.296915 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.296897 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:12.297003 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.296980 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:12.298010 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.297993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.298353 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.298337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 15:55:12.298464 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.298442 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.298519 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.298445 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.298519 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.298484 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 15:55:12.298617 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.298455 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zkn26\"" Jul 10 15:55:12.299162 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.299150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.300863 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.300841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.301478 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301456 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.301582 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301538 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fg4m9\"" Jul 10 15:55:12.301641 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jz8n7\"" Jul 10 15:55:12.301722 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301707 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 15:55:12.301765 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301706 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.301811 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301801 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.302003 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.301976 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.302156 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.302138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.302636 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.302618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 15:55:12.303504 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.303486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.303579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.303552 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:12.304289 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.304212 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.304289 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.304286 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.304441 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.304332 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wwtq4\"" Jul 10 15:55:12.304501 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.304469 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 15:55:12.304555 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.304527 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 15:55:12.304675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.304659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.305043 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.305028 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f66gp\"" Jul 10 15:55:12.305825 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.305809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.306059 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.306044 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 15:55:12.306491 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.306456 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.306576 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.306554 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.306712 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.306674 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kk29t\"" Jul 10 15:55:12.306809 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.306745 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 15:55:12.307042 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.307028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.307651 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.307632 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.307764 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.307726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g52mv\"" Jul 10 15:55:12.307911 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.307896 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.308404 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.308379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.308956 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.308941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 15:55:12.309051 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.309035 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 15:55:12.309428 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.309417 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7b92z\"" Jul 10 15:55:12.309686 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.309674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:12.309739 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.309724 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:12.310332 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 15:55:12.310463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310407 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 15:55:12.310540 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310476 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 15:55:12.310540 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310479 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 15:55:12.310910 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310893 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 15:55:12.310984 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310896 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7bhng\"" Jul 10 15:55:12.310984 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.310927 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 15:55:12.311879 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.311862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-slash\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.311968 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.311891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-socket-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.311968 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.311928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.311968 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.311962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.312080 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.311984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-os-release\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312080 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-cni-bin\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312080 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-netns\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-var-lib-kubelet\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.312202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d04291d-d190-4473-95ad-f08630129020-host-slash\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.312202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-socket-dir-parent\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-daemon-config\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-run-netns\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-env-overrides\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-etc-kubernetes\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-tuned\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-etc-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-cni-multus\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbr4\" (UniqueName: \"kubernetes.io/projected/fc4cbeda-9ded-46f4-96fd-3222264a0097-kube-api-access-xzbr4\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-lib-modules\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-var-lib-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-registration-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6pw\" (UniqueName: \"kubernetes.io/projected/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-kube-api-access-zv6pw\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-cni-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz955\" (UniqueName: \"kubernetes.io/projected/4658a983-df69-4def-af97-43eddbebdf9f-kube-api-access-lz955\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.312675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cnibin\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-k8s-cni-cncf-io\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-hostroot\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-conf-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a5080f75-77f9-4f40-9a90-f8a5cbd4bc96-konnectivity-ca\") pod \"konnectivity-agent-g9mg9\" (UID: \"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96\") " pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-kubelet\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-node-log\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97dbbccd-be38-4062-ac31-8ea70e6e6991-host\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58620b37-9b5a-40fb-b71b-d2c06e229147-kubelet-config\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-host\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-systemd-units\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.312948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-log-socket\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.312982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97dbbccd-be38-4062-ac31-8ea70e6e6991-serviceca\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-system-cni-dir\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58620b37-9b5a-40fb-b71b-d2c06e229147-dbus\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc4cbeda-9ded-46f4-96fd-3222264a0097-hosts-file\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-device-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwjr\" (UniqueName: \"kubernetes.io/projected/8d04291d-d190-4473-95ad-f08630129020-kube-api-access-9mwjr\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccace15d-9359-42f7-9e39-fc528499f3ca-tmp\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-cni-netd\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovnkube-script-lib\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-sys-fs\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-system-cni-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-cni-bin\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-kubelet\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.313424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fc8\" (UniqueName: \"kubernetes.io/projected/021357b6-f396-4532-bf9c-a5c58362b9e4-kube-api-access-j8fc8\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysctl-d\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4cbeda-9ded-46f4-96fd-3222264a0097-tmp-dir\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-run\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-cnibin\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-modprobe-d\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwn2\" (UniqueName: \"kubernetes.io/projected/ccace15d-9359-42f7-9e39-fc528499f3ca-kube-api-access-ddwn2\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d04291d-d190-4473-95ad-f08630129020-iptables-alerter-script\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-multus-certs\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-kubernetes\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysctl-conf\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-systemd\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a5080f75-77f9-4f40-9a90-f8a5cbd4bc96-agent-certs\") pod \"konnectivity-agent-g9mg9\" (UID: \"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96\") " pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.314140 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-ovn\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovnkube-config\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysconfig\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-sys\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-systemd\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.313993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8s2\" (UniqueName: \"kubernetes.io/projected/0e9aeccc-b117-4783-8c2d-09c138f839e6-kube-api-access-cg8s2\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.314017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqf9\" (UniqueName: \"kubernetes.io/projected/97dbbccd-be38-4062-ac31-8ea70e6e6991-kube-api-access-lcqf9\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.314062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-os-release\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.314594 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.314105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021357b6-f396-4532-bf9c-a5c58362b9e4-cni-binary-copy\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.318633 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.318615 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 15:55:12.323643 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.323597 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 15:50:11 +0000 UTC" deadline="2028-02-03 18:37:51.00059453 +0000 UTC" Jul 10 15:55:12.323643 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.323640 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13754h42m38.676957s" Jul 10 15:55:12.344086 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.344040 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgg7l" Jul 10 15:55:12.351150 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.351127 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgg7l" Jul 10 15:55:12.408137 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.408112 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 15:55:12.414560 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovnkube-config\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysconfig\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-sys\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-systemd\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8s2\" (UniqueName: \"kubernetes.io/projected/0e9aeccc-b117-4783-8c2d-09c138f839e6-kube-api-access-cg8s2\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysconfig\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqf9\" (UniqueName: \"kubernetes.io/projected/97dbbccd-be38-4062-ac31-8ea70e6e6991-kube-api-access-lcqf9\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-systemd\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.414714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-os-release\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021357b6-f396-4532-bf9c-a5c58362b9e4-cni-binary-copy\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-slash\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-socket-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-os-release\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-slash\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-cni-bin\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-netns\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-os-release\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-var-lib-kubelet\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d04291d-d190-4473-95ad-f08630129020-host-slash\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-os-release\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-socket-dir-parent\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-daemon-config\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.415271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-run-netns\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-env-overrides\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-cni-bin\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415101 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-etc-kubernetes\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-netns\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-tuned\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-var-lib-kubelet\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-etc-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-cni-multus\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.414861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-sys\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbr4\" (UniqueName: \"kubernetes.io/projected/fc4cbeda-9ded-46f4-96fd-3222264a0097-kube-api-access-xzbr4\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-lib-modules\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.416126 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-var-lib-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-etc-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d04291d-d190-4473-95ad-f08630129020-host-slash\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovnkube-config\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021357b6-f396-4532-bf9c-a5c58362b9e4-cni-binary-copy\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-etc-kubernetes\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-run-netns\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415579 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-socket-dir-parent\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-registration-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-lib-modules\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-cni-multus\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6pw\" (UniqueName: \"kubernetes.io/projected/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-kube-api-access-zv6pw\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clz9b\" (UniqueName: \"kubernetes.io/projected/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-kube-api-access-clz9b\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-cni-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz955\" (UniqueName: \"kubernetes.io/projected/4658a983-df69-4def-af97-43eddbebdf9f-kube-api-access-lz955\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cnibin\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-cni-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-k8s-cni-cncf-io\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cnibin\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-hostroot\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-var-lib-openvswitch\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.415097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-socket-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-env-overrides\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416126 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-hostroot\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-registration-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-k8s-cni-cncf-io\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-conf-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a5080f75-77f9-4f40-9a90-f8a5cbd4bc96-konnectivity-ca\") pod \"konnectivity-agent-g9mg9\" (UID: \"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96\") " pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-kubelet\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-conf-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-node-log\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.417857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416359 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-kubelet\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97dbbccd-be38-4062-ac31-8ea70e6e6991-host\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-node-log\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58620b37-9b5a-40fb-b71b-d2c06e229147-kubelet-config\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-host\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97dbbccd-be38-4062-ac31-8ea70e6e6991-host\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-systemd-units\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-host\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58620b37-9b5a-40fb-b71b-d2c06e229147-kubelet-config\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-log-socket\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-systemd-units\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-log-socket\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97dbbccd-be38-4062-ac31-8ea70e6e6991-serviceca\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-system-cni-dir\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416814 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-system-cni-dir\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021357b6-f396-4532-bf9c-a5c58362b9e4-multus-daemon-config\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.418783 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58620b37-9b5a-40fb-b71b-d2c06e229147-dbus\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc4cbeda-9ded-46f4-96fd-3222264a0097-hosts-file\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-device-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwjr\" (UniqueName: \"kubernetes.io/projected/8d04291d-d190-4473-95ad-f08630129020-kube-api-access-9mwjr\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccace15d-9359-42f7-9e39-fc528499f3ca-tmp\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-cni-netd\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovnkube-script-lib\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-sys-fs\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-host-cni-netd\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.416903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a5080f75-77f9-4f40-9a90-f8a5cbd4bc96-konnectivity-ca\") pod \"konnectivity-agent-g9mg9\" (UID: \"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96\") " pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-system-cni-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-cni-bin\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-kubelet\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-device-dir\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fc8\" (UniqueName: \"kubernetes.io/projected/021357b6-f396-4532-bf9c-a5c58362b9e4-kube-api-access-j8fc8\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.419679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-system-cni-dir\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97dbbccd-be38-4062-ac31-8ea70e6e6991-serviceca\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4658a983-df69-4def-af97-43eddbebdf9f-sys-fs\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58620b37-9b5a-40fb-b71b-d2c06e229147-dbus\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-kubelet\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-var-lib-cni-bin\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc4cbeda-9ded-46f4-96fd-3222264a0097-hosts-file\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.417908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovnkube-script-lib\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-tuned\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418722 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysctl-d\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4cbeda-9ded-46f4-96fd-3222264a0097-tmp-dir\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-run\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-cnibin\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-modprobe-d\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.420822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwn2\" (UniqueName: \"kubernetes.io/projected/ccace15d-9359-42f7-9e39-fc528499f3ca-kube-api-access-ddwn2\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.418998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d04291d-d190-4473-95ad-f08630129020-iptables-alerter-script\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-run\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-multus-certs\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-kubernetes\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysctl-conf\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-systemd\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysctl-d\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a5080f75-77f9-4f40-9a90-f8a5cbd4bc96-agent-certs\") pod \"konnectivity-agent-g9mg9\" (UID: \"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96\") " pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-ovn\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e9aeccc-b117-4783-8c2d-09c138f839e6-run-ovn\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4cbeda-9ded-46f4-96fd-3222264a0097-tmp-dir\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.419336 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e9aeccc-b117-4783-8c2d-09c138f839e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-cnibin\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-modprobe-d\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.419432 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:12.919393651 +0000 UTC m=+1.990211725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:12.421824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021357b6-f396-4532-bf9c-a5c58362b9e4-host-run-multus-certs\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-sysctl-conf\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-kubernetes\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccace15d-9359-42f7-9e39-fc528499f3ca-etc-systemd\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccace15d-9359-42f7-9e39-fc528499f3ca-tmp\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.419809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d04291d-d190-4473-95ad-f08630129020-iptables-alerter-script\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.420000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.421994 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.422058 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.422074 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.422150 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:55:12.922130036 +0000 UTC m=+1.992948104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:12.423160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.422203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a5080f75-77f9-4f40-9a90-f8a5cbd4bc96-agent-certs\") pod \"konnectivity-agent-g9mg9\" (UID: \"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96\") " pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.423948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.423796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6pw\" (UniqueName: \"kubernetes.io/projected/7a72ab5e-421b-4971-a437-cd2ebd9a5ff4-kube-api-access-zv6pw\") pod \"multus-additional-cni-plugins-hkh5t\" (UID: \"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4\") " pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.424294 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.424271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8s2\" (UniqueName: \"kubernetes.io/projected/0e9aeccc-b117-4783-8c2d-09c138f839e6-kube-api-access-cg8s2\") pod \"ovnkube-node-ssxhb\" (UID: \"0e9aeccc-b117-4783-8c2d-09c138f839e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.425041 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.425010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbr4\" (UniqueName: \"kubernetes.io/projected/fc4cbeda-9ded-46f4-96fd-3222264a0097-kube-api-access-xzbr4\") pod \"node-resolver-bwg7j\" (UID: \"fc4cbeda-9ded-46f4-96fd-3222264a0097\") " pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.426548 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.426526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fc8\" (UniqueName: \"kubernetes.io/projected/021357b6-f396-4532-bf9c-a5c58362b9e4-kube-api-access-j8fc8\") pod \"multus-hrjh6\" (UID: \"021357b6-f396-4532-bf9c-a5c58362b9e4\") " pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.426692 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.426581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz955\" (UniqueName: \"kubernetes.io/projected/4658a983-df69-4def-af97-43eddbebdf9f-kube-api-access-lz955\") pod \"aws-ebs-csi-driver-node-4xkrv\" (UID: \"4658a983-df69-4def-af97-43eddbebdf9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.426803 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.426786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwjr\" (UniqueName: \"kubernetes.io/projected/8d04291d-d190-4473-95ad-f08630129020-kube-api-access-9mwjr\") pod \"iptables-alerter-pgtjw\" (UID: \"8d04291d-d190-4473-95ad-f08630129020\") " pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.426803 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.426796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqf9\" (UniqueName: \"kubernetes.io/projected/97dbbccd-be38-4062-ac31-8ea70e6e6991-kube-api-access-lcqf9\") pod \"node-ca-28vk6\" (UID: \"97dbbccd-be38-4062-ac31-8ea70e6e6991\") " pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.427965 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.427948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwn2\" (UniqueName: \"kubernetes.io/projected/ccace15d-9359-42f7-9e39-fc528499f3ca-kube-api-access-ddwn2\") pod \"tuned-c8pxv\" (UID: \"ccace15d-9359-42f7-9e39-fc528499f3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.455101 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.455065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" Jul 10 15:55:12.469293 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.469268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:12.475059 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.475039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:12.520468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.520374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:12.520468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.520414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clz9b\" (UniqueName: \"kubernetes.io/projected/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-kube-api-access-clz9b\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:12.520686 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.520502 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:12.520686 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.520571 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:55:13.020551204 +0000 UTC m=+2.091369266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:12.528714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.528684 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clz9b\" (UniqueName: \"kubernetes.io/projected/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-kube-api-access-clz9b\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:12.575104 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.575061 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf23fa6bfd969eb2ea8a7d64ea7106f7.slice/crio-24a265bba3bc72434aabebb93b5f55dc0bde885e2c9cdc348916cff2187830af WatchSource:0}: Error finding container 24a265bba3bc72434aabebb93b5f55dc0bde885e2c9cdc348916cff2187830af: Status 404 returned error can't find the container with id 24a265bba3bc72434aabebb93b5f55dc0bde885e2c9cdc348916cff2187830af Jul 10 15:55:12.575490 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.575470 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6479ce33261f7d671487597879e80661.slice/crio-e7568d1df1f8e0ca746bfc7623fd167b43bf2ac1a5ff82f740f8dd5105d47912 WatchSource:0}: Error finding container e7568d1df1f8e0ca746bfc7623fd167b43bf2ac1a5ff82f740f8dd5105d47912: Status 404 returned error can't find the container with id e7568d1df1f8e0ca746bfc7623fd167b43bf2ac1a5ff82f740f8dd5105d47912 Jul 10 15:55:12.581612 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.581592 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 15:55:12.625042 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.625008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hrjh6" Jul 10 15:55:12.634881 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.634851 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021357b6_f396_4532_bf9c_a5c58362b9e4.slice/crio-449c825f8b6080169ceeeff84301addda4d2722f75623880d80bc113f738d341 WatchSource:0}: Error finding container 449c825f8b6080169ceeeff84301addda4d2722f75623880d80bc113f738d341: Status 404 returned error can't find the container with id 449c825f8b6080169ceeeff84301addda4d2722f75623880d80bc113f738d341 Jul 10 15:55:12.637902 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.637887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bwg7j" Jul 10 15:55:12.646413 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.646185 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4cbeda_9ded_46f4_96fd_3222264a0097.slice/crio-52b9c9e6a02908a77ea9a3524d39830fefdb1904f28afad9ff9209743a672ce0 WatchSource:0}: Error finding container 52b9c9e6a02908a77ea9a3524d39830fefdb1904f28afad9ff9209743a672ce0: Status 404 returned error can't find the container with id 52b9c9e6a02908a77ea9a3524d39830fefdb1904f28afad9ff9209743a672ce0 Jul 10 15:55:12.653536 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.653517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-28vk6" Jul 10 15:55:12.661757 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.661737 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dbbccd_be38_4062_ac31_8ea70e6e6991.slice/crio-4077e21309da8f059b2cf7e22f643f192ce600786d22da778065b6eccb5fed97 WatchSource:0}: Error finding container 4077e21309da8f059b2cf7e22f643f192ce600786d22da778065b6eccb5fed97: Status 404 returned error can't find the container with id 4077e21309da8f059b2cf7e22f643f192ce600786d22da778065b6eccb5fed97 Jul 10 15:55:12.676802 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.676776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" Jul 10 15:55:12.685441 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.685409 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4658a983_df69_4def_af97_43eddbebdf9f.slice/crio-9c909d218f6afa73e1e4c6628c85ef23ecb7913ddebc9df76f4ed9384d988661 WatchSource:0}: Error finding container 9c909d218f6afa73e1e4c6628c85ef23ecb7913ddebc9df76f4ed9384d988661: Status 404 returned error can't find the container with id 9c909d218f6afa73e1e4c6628c85ef23ecb7913ddebc9df76f4ed9384d988661 Jul 10 15:55:12.694006 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.693986 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 15:55:12.697782 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.697759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" Jul 10 15:55:12.707337 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.707308 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a72ab5e_421b_4971_a437_cd2ebd9a5ff4.slice/crio-48543dbc82f41c51c1875a242d4f41e921a864e0e036dd10f5fa70b9bf00feca WatchSource:0}: Error finding container 48543dbc82f41c51c1875a242d4f41e921a864e0e036dd10f5fa70b9bf00feca: Status 404 returned error can't find the container with id 48543dbc82f41c51c1875a242d4f41e921a864e0e036dd10f5fa70b9bf00feca Jul 10 15:55:12.715920 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.715887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pgtjw" Jul 10 15:55:12.724969 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.724944 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d04291d_d190_4473_95ad_f08630129020.slice/crio-e053afad8bfb7c55371b43e8ac088e5908514cef72ceba26b827a8e9cadc7773 WatchSource:0}: Error finding container e053afad8bfb7c55371b43e8ac088e5908514cef72ceba26b827a8e9cadc7773: Status 404 returned error can't find the container with id e053afad8bfb7c55371b43e8ac088e5908514cef72ceba26b827a8e9cadc7773 Jul 10 15:55:12.923298 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.923196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:12.923457 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:12.923300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:12.923457 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.923384 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:12.923457 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.923428 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:12.923457 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.923433 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:12.923457 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.923448 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:12.923705 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.923544 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:13.923477909 +0000 UTC m=+2.994295971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:12.923705 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:12.923574 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:55:13.923563046 +0000 UTC m=+2.994381101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:12.941376 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:12.941350 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9aeccc_b117_4783_8c2d_09c138f839e6.slice/crio-3b02eb713445cc284b056f9fd34f331d9918178288c213433f5090c8592ceb95 WatchSource:0}: Error finding container 3b02eb713445cc284b056f9fd34f331d9918178288c213433f5090c8592ceb95: Status 404 returned error can't find the container with id 3b02eb713445cc284b056f9fd34f331d9918178288c213433f5090c8592ceb95 Jul 10 15:55:13.024529 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.024489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:13.024697 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.024678 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:13.024773 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.024759 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:55:14.024738794 +0000 UTC m=+3.095556844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:13.028929 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:13.028900 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccace15d_9359_42f7_9e39_fc528499f3ca.slice/crio-4c40d597a8f9242942634a00959de0f444566ffa9f43b5ef668dca2b127ab663 WatchSource:0}: Error finding container 4c40d597a8f9242942634a00959de0f444566ffa9f43b5ef668dca2b127ab663: Status 404 returned error can't find the container with id 4c40d597a8f9242942634a00959de0f444566ffa9f43b5ef668dca2b127ab663 Jul 10 15:55:13.320773 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.320736 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 15:55:13.353703 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.353642 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 15:50:12 +0000 UTC" deadline="2028-01-05 07:07:53.441333121 +0000 UTC" Jul 10 15:55:13.353703 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.353700 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13047h12m40.0876368s" Jul 10 15:55:13.495977 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.495939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:13.496115 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.496056 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:13.496582 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.496443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:13.496582 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.496531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:13.498972 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.498853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" event={"ID":"4658a983-df69-4def-af97-43eddbebdf9f","Type":"ContainerStarted","Data":"9c909d218f6afa73e1e4c6628c85ef23ecb7913ddebc9df76f4ed9384d988661"} Jul 10 15:55:13.508716 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.508628 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-28vk6" event={"ID":"97dbbccd-be38-4062-ac31-8ea70e6e6991","Type":"ContainerStarted","Data":"4077e21309da8f059b2cf7e22f643f192ce600786d22da778065b6eccb5fed97"} Jul 10 15:55:13.511397 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.511363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bwg7j" event={"ID":"fc4cbeda-9ded-46f4-96fd-3222264a0097","Type":"ContainerStarted","Data":"52b9c9e6a02908a77ea9a3524d39830fefdb1904f28afad9ff9209743a672ce0"} Jul 10 15:55:13.519818 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.519762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hrjh6" event={"ID":"021357b6-f396-4532-bf9c-a5c58362b9e4","Type":"ContainerStarted","Data":"449c825f8b6080169ceeeff84301addda4d2722f75623880d80bc113f738d341"} Jul 10 15:55:13.529704 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.529664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g9mg9" event={"ID":"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96","Type":"ContainerStarted","Data":"07fd06575bb79f0a385110236aab888edb3edcc7f80df81c155ad1d2ef969d58"} Jul 10 15:55:13.539048 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.539012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" event={"ID":"ccace15d-9359-42f7-9e39-fc528499f3ca","Type":"ContainerStarted","Data":"4c40d597a8f9242942634a00959de0f444566ffa9f43b5ef668dca2b127ab663"} Jul 10 15:55:13.544003 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.543957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"3b02eb713445cc284b056f9fd34f331d9918178288c213433f5090c8592ceb95"} Jul 10 15:55:13.549070 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.549036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pgtjw" event={"ID":"8d04291d-d190-4473-95ad-f08630129020","Type":"ContainerStarted","Data":"e053afad8bfb7c55371b43e8ac088e5908514cef72ceba26b827a8e9cadc7773"} Jul 10 15:55:13.554314 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.554284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerStarted","Data":"48543dbc82f41c51c1875a242d4f41e921a864e0e036dd10f5fa70b9bf00feca"} Jul 10 15:55:13.562229 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.562197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" event={"ID":"af23fa6bfd969eb2ea8a7d64ea7106f7","Type":"ContainerStarted","Data":"24a265bba3bc72434aabebb93b5f55dc0bde885e2c9cdc348916cff2187830af"} Jul 10 15:55:13.564892 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.564863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" event={"ID":"6479ce33261f7d671487597879e80661","Type":"ContainerStarted","Data":"e7568d1df1f8e0ca746bfc7623fd167b43bf2ac1a5ff82f740f8dd5105d47912"} Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.934744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:13.934875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.935016 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.935084 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:15.935065925 +0000 UTC m=+5.005884008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.935175 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.935190 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.935203 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:13.935579 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:13.935237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:55:15.935225998 +0000 UTC m=+5.006044060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:14.036348 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:14.035587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:14.036348 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:14.035858 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:14.036348 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:14.035925 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:55:16.035903899 +0000 UTC m=+5.106721948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:14.354278 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:14.354218 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 15:50:12 +0000 UTC" deadline="2028-03-30 21:19:19.18940778 +0000 UTC" Jul 10 15:55:14.354278 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:14.354276 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15101h24m4.835137322s" Jul 10 15:55:14.488971 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:14.488932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:14.489163 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:14.489080 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:15.509184 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:15.508938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:15.509184 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.509061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:15.509715 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:15.509444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:15.509715 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.509538 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:15.955480 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:15.955442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:15.955668 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:15.955506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:15.955668 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.955630 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:15.955668 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.955649 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:15.955668 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.955663 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:15.955880 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.955719 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:55:19.955700589 +0000 UTC m=+9.026518641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:15.956131 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.956108 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:15.956227 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:15.956189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:19.956152131 +0000 UTC m=+9.026970181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:16.056669 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:16.056574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:16.056855 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:16.056773 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:16.056855 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:16.056841 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:55:20.056822918 +0000 UTC m=+9.127640981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:16.489226 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:16.489195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:16.489426 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:16.489360 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:17.489451 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:17.489419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:17.489882 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:17.489522 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:17.489882 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:17.489815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:17.489882 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:17.489873 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:18.489371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:18.489337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:18.489561 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:18.489480 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:19.489401 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:19.489363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:19.489593 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.489510 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:19.489982 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:19.489960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:19.490075 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.490062 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:19.994824 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:19.994780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:19.995017 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:19.994854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:19.995017 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.995011 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:19.995109 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.995031 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:19.995109 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.995044 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:19.995109 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.995096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:55:27.995083033 +0000 UTC m=+17.065901081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:19.995429 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.995403 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:19.995527 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:19.995483 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:27.995464459 +0000 UTC m=+17.066282509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:20.096166 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:20.096130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:20.096382 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:20.096340 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:20.096449 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:20.096407 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:55:28.096389216 +0000 UTC m=+17.167207264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:20.489005 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:20.488970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:20.489192 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:20.489099 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:21.489582 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:21.489551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:21.490048 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:21.489635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:21.490048 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:21.489682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:21.490048 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:21.489721 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:22.489768 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:22.489738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:22.490176 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:22.489872 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:23.493414 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:23.493379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:23.493847 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:23.493501 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:23.493934 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:23.493866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:23.493987 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:23.493955 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:24.489540 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:24.489502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:24.489722 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:24.489644 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:25.489035 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:25.488996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:25.489468 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:25.489120 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:25.489468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:25.489157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:25.489468 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:25.489231 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:26.489891 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:26.489848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:26.490335 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:26.490003 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:27.489129 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:27.489091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:27.489341 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:27.489146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:27.489341 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:27.489244 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:27.489467 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:27.489364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:28.055648 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:28.055601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:28.055665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.055789 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.055804 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.055822 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.055836 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.055863 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:44.055841904 +0000 UTC m=+33.126659960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:28.056122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.055883 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:55:44.055872467 +0000 UTC m=+33.126690515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:28.156544 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:28.156500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:28.156713 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.156640 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:28.156774 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.156716 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:55:44.156694278 +0000 UTC m=+33.227512329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:28.489516 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:28.489483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:28.489687 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:28.489593 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:29.489828 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:29.489794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:29.490295 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:29.489907 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:29.490295 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:29.489965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:29.490295 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:29.490052 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:30.489229 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:30.489193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:30.489434 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:30.489383 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:31.490449 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:31.490417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:31.490889 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:31.490576 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:31.490889 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:31.490651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:31.490889 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:31.490811 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:32.489312 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:32.489271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:32.489460 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:32.489404 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:33.489855 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.489404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:33.489855 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:33.489651 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:33.489855 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.489682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:33.489855 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:33.489781 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:33.611098 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.610873 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" event={"ID":"af23fa6bfd969eb2ea8a7d64ea7106f7","Type":"ContainerStarted","Data":"ce19a5a20fceb51b600dda713d081ad499d2127152c911047589e8a659f9b9a9"} Jul 10 15:55:33.612320 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.612237 2576 generic.go:356] "Generic (PLEG): container finished" podID="6479ce33261f7d671487597879e80661" containerID="d027f621b6e6e1dcd4d1b69341c494dcb5891c8812eeac7ba65fd3ee66d5eec7" exitCode=0 Jul 10 15:55:33.612320 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.612289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" event={"ID":"6479ce33261f7d671487597879e80661","Type":"ContainerDied","Data":"d027f621b6e6e1dcd4d1b69341c494dcb5891c8812eeac7ba65fd3ee66d5eec7"} Jul 10 15:55:33.613547 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.613528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" event={"ID":"4658a983-df69-4def-af97-43eddbebdf9f","Type":"ContainerStarted","Data":"b7c80886487c8721a604bebf46d1c936080c3cd7a80549b7941c7444c338545e"} Jul 10 15:55:33.614773 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.614755 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-28vk6" event={"ID":"97dbbccd-be38-4062-ac31-8ea70e6e6991","Type":"ContainerStarted","Data":"ab24b972b4e9adf9b267543c2b289022c6d21f4aee00a7666585dd48cc3dc6e2"} Jul 10 15:55:33.616020 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.615996 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bwg7j" event={"ID":"fc4cbeda-9ded-46f4-96fd-3222264a0097","Type":"ContainerStarted","Data":"cf4f4f9d42be740a7588e2ec36b7576acceea019ff3018ad61d01f139b2956c5"} Jul 10 15:55:33.617206 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.617181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hrjh6" event={"ID":"021357b6-f396-4532-bf9c-a5c58362b9e4","Type":"ContainerStarted","Data":"cf4b9cec55f1b6cd44bfd927e16536eaa7befb7fef0eabeb622a0b32a9ebcc5c"} Jul 10 15:55:33.618653 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.618633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g9mg9" event={"ID":"a5080f75-77f9-4f40-9a90-f8a5cbd4bc96","Type":"ContainerStarted","Data":"7e06647905f5830f3ceb1a939d5432a9b35750fd11ab1c6b2e6c5027d12173ee"} Jul 10 15:55:33.620170 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.620150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" event={"ID":"ccace15d-9359-42f7-9e39-fc528499f3ca","Type":"ContainerStarted","Data":"20507a1224799406383cd25fb4aa2f4ef0a1c9f552f6c9c4406e09d024da753a"} Jul 10 15:55:33.622144 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.622127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 15:55:33.622440 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.622423 2576 generic.go:356] "Generic (PLEG): container finished" podID="0e9aeccc-b117-4783-8c2d-09c138f839e6" containerID="a873dbb57b3e1f554bbafd6c1d77c7134a8a2682563fc5201ab6f01d2722e497" exitCode=1 Jul 10 15:55:33.622494 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.622478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"b377988087ee608f7c2401d53277031c02c584469047aafe846f3c136534e036"} Jul 10 15:55:33.622541 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.622526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"04e802b1ae5b75a3ea2ab88d90fbaf111359b2f5ba29cd72fab566306b948519"} Jul 10 15:55:33.622586 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.622556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerDied","Data":"a873dbb57b3e1f554bbafd6c1d77c7134a8a2682563fc5201ab6f01d2722e497"} Jul 10 15:55:33.622622 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.622592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"7ec5a97ce6d3c69139e15803b2a5cde7f62e7778f5c083a99a5f2df92501920e"} Jul 10 15:55:33.623716 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.623698 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a72ab5e-421b-4971-a437-cd2ebd9a5ff4" containerID="0955ba8c587a4110567c73d9b6ea00d034ed66a139e1790076f1a2986efdcded" exitCode=0 Jul 10 15:55:33.623785 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.623723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerDied","Data":"0955ba8c587a4110567c73d9b6ea00d034ed66a139e1790076f1a2986efdcded"} Jul 10 15:55:33.646409 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.646362 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-69.ec2.internal" podStartSLOduration=21.646347429 podStartE2EDuration="21.646347429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:33.629151911 +0000 UTC m=+22.699969981" watchObservedRunningTime="2026-07-10 15:55:33.646347429 +0000 UTC m=+22.717165499" Jul 10 15:55:33.646672 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.646649 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c8pxv" podStartSLOduration=2.619827786 podStartE2EDuration="22.646642454s" totalImagesPullingTime="20.026814668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:33.646017664 +0000 UTC m=+22.716835733" watchObservedRunningTime="2026-07-10 15:55:33.646642454 +0000 UTC m=+22.717460525" Jul 10 15:55:33.660199 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.660158 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g9mg9" podStartSLOduration=3.304354656 podStartE2EDuration="22.660143643s" totalImagesPullingTime="19.355788987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:33.659882166 +0000 UTC m=+22.730700236" watchObservedRunningTime="2026-07-10 15:55:33.660143643 +0000 UTC m=+22.730961712" Jul 10 15:55:33.677415 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.677373 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hrjh6" podStartSLOduration=2.150920629 podStartE2EDuration="22.677358728s" totalImagesPullingTime="20.526438099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:33.677104847 +0000 UTC m=+22.747922919" watchObservedRunningTime="2026-07-10 15:55:33.677358728 +0000 UTC m=+22.748176825" Jul 10 15:55:33.692891 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.692842 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bwg7j" podStartSLOduration=2.512511626 podStartE2EDuration="22.692826498s" totalImagesPullingTime="20.180314872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:33.692655686 +0000 UTC m=+22.763473755" watchObservedRunningTime="2026-07-10 15:55:33.692826498 +0000 UTC m=+22.763644568" Jul 10 15:55:33.707740 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:33.707698 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-28vk6" podStartSLOduration=10.943046602 podStartE2EDuration="22.707684023s" totalImagesPullingTime="11.764637421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:33.707662014 +0000 UTC m=+22.778480085" watchObservedRunningTime="2026-07-10 15:55:33.707684023 +0000 UTC m=+22.778502092" Jul 10 15:55:34.145218 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.145133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:34.145718 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.145701 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:34.488989 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.488953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:34.489181 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:34.489079 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:34.629271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.629225 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 15:55:34.629822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.629661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"d0985da73159cf160d9173d78e8626e2be19ec98a37c3d63d376c053d8e45055"} Jul 10 15:55:34.629822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.629748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"963dff800597315ec66e7e76109fa38c7ee0b0ac75d1e32660038ca9bcce92c8"} Jul 10 15:55:34.631330 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.631294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pgtjw" event={"ID":"8d04291d-d190-4473-95ad-f08630129020","Type":"ContainerStarted","Data":"d4f7dcb7d260f9daddc3fea9827aff9ae3a25df834cd591c185aaf4b36055636"} Jul 10 15:55:34.649373 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:34.649287 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pgtjw" podStartSLOduration=3.547674957 podStartE2EDuration="23.649246657s" totalImagesPullingTime="20.1015717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:34.648890218 +0000 UTC m=+23.719708288" watchObservedRunningTime="2026-07-10 15:55:34.649246657 +0000 UTC m=+23.720064729" Jul 10 15:55:35.023609 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.023445 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 15:55:35.386462 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.386316 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T15:55:35.023601449Z","UUID":"cbf3aeba-3f60-4502-bbe9-5ae7150f9df7","Handler":null,"Name":"","Endpoint":""} Jul 10 15:55:35.387870 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.387848 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 15:55:35.388001 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.387882 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 15:55:35.489342 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.489307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:35.489518 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.489344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:35.489518 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:35.489445 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:35.489709 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:35.489579 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:35.635788 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.635749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" event={"ID":"6479ce33261f7d671487597879e80661","Type":"ContainerStarted","Data":"ba8dcbe5d5a6232ae4408e297517c275bbb1e55288016ecf9934f9e2e6d088cd"} Jul 10 15:55:35.638127 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.638066 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 15:55:35.638127 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.638077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" event={"ID":"4658a983-df69-4def-af97-43eddbebdf9f","Type":"ContainerStarted","Data":"67a37b836d4dfedfac58131457e074bef0d84fa45d1d5235223612683930bafa"} Jul 10 15:55:35.660177 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:35.660125 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-69.ec2.internal" podStartSLOduration=23.660109421 podStartE2EDuration="23.660109421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:35.659624523 +0000 UTC m=+24.730442592" watchObservedRunningTime="2026-07-10 15:55:35.660109421 +0000 UTC m=+24.730927490" Jul 10 15:55:36.489585 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:36.489499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:36.489752 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:36.489650 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:36.646227 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:36.643844 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 15:55:36.646227 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:36.644406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"5846cf3656ff49007b8fcba00e810a537c05f2eefc80dca94779b6955062011b"} Jul 10 15:55:36.695021 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:36.694983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:36.849532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:36.849446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:36.932945 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:36.932905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:37.489341 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:37.489306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:37.489341 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:37.489325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:37.489562 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:37.489434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:37.489562 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:37.489505 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:37.783567 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:37.783488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:37.884696 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:37.884662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:38.489071 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.488899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:38.489281 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:38.489166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:38.650833 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.650687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" event={"ID":"4658a983-df69-4def-af97-43eddbebdf9f","Type":"ContainerStarted","Data":"7319b355284fc04c973d494cfb2eac42933765a92295de55b9642986f4fb60f5"} Jul 10 15:55:38.653622 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.653593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 15:55:38.653925 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.653904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"1c51745d37386f2307b2adc421bd88d564055a722fe453f53a3e564716a1df45"} Jul 10 15:55:38.654196 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.654180 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:38.654370 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.654352 2576 scope.go:117] "RemoveContainer" containerID="a873dbb57b3e1f554bbafd6c1d77c7134a8a2682563fc5201ab6f01d2722e497" Jul 10 15:55:38.656008 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.655986 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a72ab5e-421b-4971-a437-cd2ebd9a5ff4" containerID="2476d4e917ea3ac33919bb7bbee789e5b6f804f5a77b3ddb3320f7210180a4ac" exitCode=0 Jul 10 15:55:38.656090 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.656015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerDied","Data":"2476d4e917ea3ac33919bb7bbee789e5b6f804f5a77b3ddb3320f7210180a4ac"} Jul 10 15:55:38.671621 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.671598 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:38.671875 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:38.671579 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xkrv" podStartSLOduration=3.118262852 podStartE2EDuration="27.671563756s" totalImagesPullingTime="24.553300904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:38.670751848 +0000 UTC m=+27.741569909" watchObservedRunningTime="2026-07-10 15:55:38.671563756 +0000 UTC m=+27.742381825" Jul 10 15:55:39.490082 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.490045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:39.490855 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.490090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:39.490855 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:39.490186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:39.490855 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:39.490269 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:39.662963 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.662937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 15:55:39.663426 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.663395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" event={"ID":"0e9aeccc-b117-4783-8c2d-09c138f839e6","Type":"ContainerStarted","Data":"f9eac0997a03a9dc679187eea35dcade7d5215c2784fdb3e80bee955e164c9da"} Jul 10 15:55:39.663588 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.663569 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:39.663665 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.663652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:39.681123 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.681093 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:55:39.697544 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:39.697468 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" podStartSLOduration=8.53801113 podStartE2EDuration="28.697444846s" totalImagesPullingTime="20.159433716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:39.696561527 +0000 UTC m=+28.767379600" watchObservedRunningTime="2026-07-10 15:55:39.697444846 +0000 UTC m=+28.768262920" Jul 10 15:55:40.293377 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.293134 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddv99"] Jul 10 15:55:40.293558 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.293448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:40.293558 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:40.293531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:40.295604 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.295574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85zw9"] Jul 10 15:55:40.295730 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.295701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:40.295816 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:40.295799 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:40.305482 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.305455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kw467"] Jul 10 15:55:40.305580 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.305545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:40.305628 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:40.305613 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:40.667833 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.667741 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a72ab5e-421b-4971-a437-cd2ebd9a5ff4" containerID="a1ebb0cd1df6ef80513df40f81d16c2dca3fc2404a02f142328243fa89c7b120" exitCode=0 Jul 10 15:55:40.668397 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:40.667823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerDied","Data":"a1ebb0cd1df6ef80513df40f81d16c2dca3fc2404a02f142328243fa89c7b120"} Jul 10 15:55:41.397076 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:41.397039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:41.397247 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:41.397151 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 15:55:41.397878 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:41.397613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g9mg9" Jul 10 15:55:41.477086 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:41.477032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:41.491906 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:41.491876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:41.492400 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:41.492369 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:41.492836 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:41.492812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:41.492954 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:41.492935 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:42.489383 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:42.489356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:42.489918 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:42.489460 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:42.674944 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:42.674907 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a72ab5e-421b-4971-a437-cd2ebd9a5ff4" containerID="f3788fc6c580be6395e661138ed7a8b50046250a2dec68e59ab720de534a6d5c" exitCode=0 Jul 10 15:55:42.675095 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:42.674964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerDied","Data":"f3788fc6c580be6395e661138ed7a8b50046250a2dec68e59ab720de534a6d5c"} Jul 10 15:55:43.493996 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:43.493964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:43.494527 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:43.494004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:43.494527 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:43.494084 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:43.494527 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:43.494176 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:44.079565 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.079361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:44.079748 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.079602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:44.079748 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.079514 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:44.079748 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.079727 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 15:55:44.079748 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.079746 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 15:55:44.079916 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.079755 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret podName:58620b37-9b5a-40fb-b71b-d2c06e229147 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:16.07972993 +0000 UTC m=+65.150547982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret") pod "global-pull-secret-syncer-kw467" (UID: "58620b37-9b5a-40fb-b71b-d2c06e229147") : object "kube-system"/"original-pull-secret" not registered Jul 10 15:55:44.079916 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.079759 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rksw9 for pod openshift-network-diagnostics/network-check-target-ddv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:44.079916 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.079810 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9 podName:e7076633-c37e-4dca-881c-1501718695db nodeName:}" failed. No retries permitted until 2026-07-10 15:56:16.079796395 +0000 UTC m=+65.150614464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rksw9" (UniqueName: "kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9") pod "network-check-target-ddv99" (UID: "e7076633-c37e-4dca-881c-1501718695db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 15:55:44.180182 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.180143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:44.180371 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.180348 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:44.180445 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.180431 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs podName:3dae35b6-cf5b-4c24-9390-da5f1e0d93fc nodeName:}" failed. No retries permitted until 2026-07-10 15:56:16.180409866 +0000 UTC m=+65.251227928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs") pod "network-metrics-daemon-85zw9" (UID: "3dae35b6-cf5b-4c24-9390-da5f1e0d93fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 15:55:44.399446 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.399354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:44.478375 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.478336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:44.489752 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.489726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:44.489893 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:44.489862 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw467" podUID="58620b37-9b5a-40fb-b71b-d2c06e229147" Jul 10 15:55:44.647648 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:44.647610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:45.492523 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.492493 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:45.492695 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:45.492613 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85zw9" podUID="3dae35b6-cf5b-4c24-9390-da5f1e0d93fc" Jul 10 15:55:45.493019 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.492999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:45.493122 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:45.493102 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddv99" podUID="e7076633-c37e-4dca-881c-1501718695db" Jul 10 15:55:45.763671 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.763604 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-69.ec2.internal" event="NodeReady" Jul 10 15:55:45.764037 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.763758 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 15:55:45.808641 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.808609 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvmrt"] Jul 10 15:55:45.811983 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.811953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.812295 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.812271 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8z8w6"] Jul 10 15:55:45.814340 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.814315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nh8x5\"" Jul 10 15:55:45.814707 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.814476 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 15:55:45.814707 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.814478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 15:55:45.815201 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.815181 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:45.817033 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.817011 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 15:55:45.817324 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.817238 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zk7gp\"" Jul 10 15:55:45.817324 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.817292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 15:55:45.817498 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.817481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 15:55:45.823468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.823445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvmrt"] Jul 10 15:55:45.825008 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.824983 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8z8w6"] Jul 10 15:55:45.893112 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.893067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-config-volume\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.893112 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.893110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.893372 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.893131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czwp\" (UniqueName: \"kubernetes.io/projected/db137799-74b2-45af-9d93-b93d5939ffdb-kube-api-access-4czwp\") pod \"ingress-canary-8z8w6\" (UID: \"db137799-74b2-45af-9d93-b93d5939ffdb\") " pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:45.893372 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.893206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx4v\" (UniqueName: \"kubernetes.io/projected/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-kube-api-access-cvx4v\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.893372 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.893303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db137799-74b2-45af-9d93-b93d5939ffdb-cert\") pod \"ingress-canary-8z8w6\" (UID: \"db137799-74b2-45af-9d93-b93d5939ffdb\") " pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:45.893372 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.893338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-tmp-dir\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.918425 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.918335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:45.994532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-config-volume\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.994532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.994752 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czwp\" (UniqueName: \"kubernetes.io/projected/db137799-74b2-45af-9d93-b93d5939ffdb-kube-api-access-4czwp\") pod \"ingress-canary-8z8w6\" (UID: \"db137799-74b2-45af-9d93-b93d5939ffdb\") " pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:45.994752 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx4v\" (UniqueName: \"kubernetes.io/projected/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-kube-api-access-cvx4v\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.994752 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db137799-74b2-45af-9d93-b93d5939ffdb-cert\") pod \"ingress-canary-8z8w6\" (UID: \"db137799-74b2-45af-9d93-b93d5939ffdb\") " pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:45.994752 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-tmp-dir\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.995542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.994938 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-tmp-dir\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:45.995542 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:45.995059 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 15:55:45.995542 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:45.995119 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls podName:5b65bd4f-cccb-4ae4-929d-e1f2331394a8 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:46.495099525 +0000 UTC m=+35.565917577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls") pod "dns-default-pvmrt" (UID: "5b65bd4f-cccb-4ae4-929d-e1f2331394a8") : secret "dns-default-metrics-tls" not found Jul 10 15:55:45.995542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:45.995185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-config-volume\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:46.000435 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.000412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db137799-74b2-45af-9d93-b93d5939ffdb-cert\") pod \"ingress-canary-8z8w6\" (UID: \"db137799-74b2-45af-9d93-b93d5939ffdb\") " pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:46.003651 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.003602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx4v\" (UniqueName: \"kubernetes.io/projected/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-kube-api-access-cvx4v\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:46.003651 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.003645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czwp\" (UniqueName: \"kubernetes.io/projected/db137799-74b2-45af-9d93-b93d5939ffdb-kube-api-access-4czwp\") pod \"ingress-canary-8z8w6\" (UID: \"db137799-74b2-45af-9d93-b93d5939ffdb\") " pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:46.088427 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.088330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:46.135486 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.135446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8z8w6" Jul 10 15:55:46.489750 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.489717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:55:46.492659 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.492638 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 15:55:46.499860 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.499840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:46.499970 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:46.499950 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 15:55:46.500010 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:46.499998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls podName:5b65bd4f-cccb-4ae4-929d-e1f2331394a8 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:47.499985551 +0000 UTC m=+36.570803600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls") pod "dns-default-pvmrt" (UID: "5b65bd4f-cccb-4ae4-929d-e1f2331394a8") : secret "dns-default-metrics-tls" not found Jul 10 15:55:46.810849 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:46.810822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8z8w6"] Jul 10 15:55:46.814918 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:46.814878 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb137799_74b2_45af_9d93_b93d5939ffdb.slice/crio-68a740b95118bba80d6e531be47a1288f8c0ecd34f82727afb078786ec6e0745 WatchSource:0}: Error finding container 68a740b95118bba80d6e531be47a1288f8c0ecd34f82727afb078786ec6e0745: Status 404 returned error can't find the container with id 68a740b95118bba80d6e531be47a1288f8c0ecd34f82727afb078786ec6e0745 Jul 10 15:55:47.489380 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.489348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:55:47.489380 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.489365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:55:47.492810 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.492782 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8rkdc\"" Jul 10 15:55:47.493025 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.493002 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 15:55:47.493096 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.493060 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 15:55:47.493226 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.493210 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 15:55:47.493311 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.493294 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2tf4w\"" Jul 10 15:55:47.507387 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.507360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:47.507569 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:47.507522 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 15:55:47.507641 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:47.507605 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls podName:5b65bd4f-cccb-4ae4-929d-e1f2331394a8 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:49.507583779 +0000 UTC m=+38.578401846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls") pod "dns-default-pvmrt" (UID: "5b65bd4f-cccb-4ae4-929d-e1f2331394a8") : secret "dns-default-metrics-tls" not found Jul 10 15:55:47.693307 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.693276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8z8w6" event={"ID":"db137799-74b2-45af-9d93-b93d5939ffdb","Type":"ContainerStarted","Data":"68a740b95118bba80d6e531be47a1288f8c0ecd34f82727afb078786ec6e0745"} Jul 10 15:55:47.696524 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.696493 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a72ab5e-421b-4971-a437-cd2ebd9a5ff4" containerID="cdba0c422922bda267c53c90fa85872f2d7187046f9f3dfc4cd1c15a63d7b08e" exitCode=0 Jul 10 15:55:47.696668 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:47.696551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerDied","Data":"cdba0c422922bda267c53c90fa85872f2d7187046f9f3dfc4cd1c15a63d7b08e"} Jul 10 15:55:48.704828 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:48.704795 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a72ab5e-421b-4971-a437-cd2ebd9a5ff4" containerID="9a36c2c6a985b87628f6b7d1e858ef632e7dd16ab5514b7ca1311bc2dd99954b" exitCode=0 Jul 10 15:55:48.705324 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:48.704855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerDied","Data":"9a36c2c6a985b87628f6b7d1e858ef632e7dd16ab5514b7ca1311bc2dd99954b"} Jul 10 15:55:49.452899 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.452871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:49.453300 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.453279 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966"] Jul 10 15:55:49.463927 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.463894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.467450 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.467426 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 15:55:49.467598 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.467576 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7rqsb\"" Jul 10 15:55:49.467598 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.467587 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 15:55:49.467709 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.467595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 15:55:49.467709 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.467587 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 15:55:49.469675 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.469655 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966"] Jul 10 15:55:49.523869 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.523801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:49.524527 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:49.524012 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 15:55:49.524527 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:49.524102 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls podName:5b65bd4f-cccb-4ae4-929d-e1f2331394a8 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:53.524082042 +0000 UTC m=+42.594900101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls") pod "dns-default-pvmrt" (UID: "5b65bd4f-cccb-4ae4-929d-e1f2331394a8") : secret "dns-default-metrics-tls" not found Jul 10 15:55:49.561489 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.561460 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv"] Jul 10 15:55:49.570365 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.570340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.571409 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.571365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:49.572916 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.572881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 15:55:49.572916 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.572892 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 15:55:49.572916 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.572900 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dld4b\"" Jul 10 15:55:49.573160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.572881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 15:55:49.573160 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.572885 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 15:55:49.581339 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.581320 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv"] Jul 10 15:55:49.624532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.624441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qlq\" (UniqueName: \"kubernetes.io/projected/573e9a41-ff70-4e11-a4f4-aa94e16e6183-kube-api-access-t7qlq\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.624532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.624505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/573e9a41-ff70-4e11-a4f4-aa94e16e6183-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.624532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.624524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.658634 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.658598 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-hzdr6"] Jul 10 15:55:49.672948 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.672922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.677084 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.677047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-hzdr6"] Jul 10 15:55:49.679841 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.678897 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 15:55:49.679841 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.679725 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 15:55:49.679841 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.679825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 15:55:49.680110 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.680001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 15:55:49.680485 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.680330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9nq2x\"" Jul 10 15:55:49.683168 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.683147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 15:55:49.709275 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.709206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8z8w6" event={"ID":"db137799-74b2-45af-9d93-b93d5939ffdb","Type":"ContainerStarted","Data":"f842ff5b875bd95e6e5a6d3cba892f7d6c7c0ea64ee792ed81f902ba9a1177c9"} Jul 10 15:55:49.712727 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.712698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" event={"ID":"7a72ab5e-421b-4971-a437-cd2ebd9a5ff4","Type":"ContainerStarted","Data":"2fa87fc725a750d9cfef14b9288d22c5ad418e125ee3f5fc5225cca63dece000"} Jul 10 15:55:49.719230 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.719198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:49.725146 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/573e9a41-ff70-4e11-a4f4-aa94e16e6183-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.725328 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.725328 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a95360d0-7c80-4f54-a510-47b017ebc24e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.725328 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a95360d0-7c80-4f54-a510-47b017ebc24e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.725328 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qlq\" (UniqueName: \"kubernetes.io/projected/573e9a41-ff70-4e11-a4f4-aa94e16e6183-kube-api-access-t7qlq\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.725328 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:49.725314 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:49.725596 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724n6\" (UniqueName: \"kubernetes.io/projected/a95360d0-7c80-4f54-a510-47b017ebc24e-kube-api-access-724n6\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.725596 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:49.725376 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls podName:573e9a41-ff70-4e11-a4f4-aa94e16e6183 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:50.225356208 +0000 UTC m=+39.296174264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wk966" (UID: "573e9a41-ff70-4e11-a4f4-aa94e16e6183") : secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:49.725895 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.725872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/573e9a41-ff70-4e11-a4f4-aa94e16e6183-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.729035 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.728074 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8z8w6" podStartSLOduration=2.040645736 podStartE2EDuration="4.728056257s" totalImagesPullingTime="2.687410521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:49.72760117 +0000 UTC m=+38.798419242" watchObservedRunningTime="2026-07-10 15:55:49.728056257 +0000 UTC m=+38.798874306" Jul 10 15:55:49.747496 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.747465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qlq\" (UniqueName: \"kubernetes.io/projected/573e9a41-ff70-4e11-a4f4-aa94e16e6183-kube-api-access-t7qlq\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:49.764306 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.764241 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hkh5t" podStartSLOduration=7.269520417 podStartE2EDuration="38.764225574s" totalImagesPullingTime="31.494705157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:49.762727701 +0000 UTC m=+38.833545771" watchObservedRunningTime="2026-07-10 15:55:49.764225574 +0000 UTC m=+38.835043643" Jul 10 15:55:49.819672 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.819634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:49.826581 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef6f967d-0657-47e5-bb08-efaffbaca4fa-config\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.826711 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a95360d0-7c80-4f54-a510-47b017ebc24e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.826711 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f967d-0657-47e5-bb08-efaffbaca4fa-trusted-ca\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.826811 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a95360d0-7c80-4f54-a510-47b017ebc24e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.826871 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef6f967d-0657-47e5-bb08-efaffbaca4fa-serving-cert\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.826923 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blb5s\" (UniqueName: \"kubernetes.io/projected/ef6f967d-0657-47e5-bb08-efaffbaca4fa-kube-api-access-blb5s\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.826985 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.826934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724n6\" (UniqueName: \"kubernetes.io/projected/a95360d0-7c80-4f54-a510-47b017ebc24e-kube-api-access-724n6\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.827204 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.827180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a95360d0-7c80-4f54-a510-47b017ebc24e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.829163 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.829145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a95360d0-7c80-4f54-a510-47b017ebc24e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.835564 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.835543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-724n6\" (UniqueName: \"kubernetes.io/projected/a95360d0-7c80-4f54-a510-47b017ebc24e-kube-api-access-724n6\") pod \"kube-storage-version-migrator-operator-799cb7f958-7dtpv\" (UID: \"a95360d0-7c80-4f54-a510-47b017ebc24e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.879624 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.879557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" Jul 10 15:55:49.921044 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.921011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:49.927301 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.927267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef6f967d-0657-47e5-bb08-efaffbaca4fa-serving-cert\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.927452 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.927327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blb5s\" (UniqueName: \"kubernetes.io/projected/ef6f967d-0657-47e5-bb08-efaffbaca4fa-kube-api-access-blb5s\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.927452 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.927372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef6f967d-0657-47e5-bb08-efaffbaca4fa-config\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.927452 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.927436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f967d-0657-47e5-bb08-efaffbaca4fa-trusted-ca\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.928032 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.928006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef6f967d-0657-47e5-bb08-efaffbaca4fa-config\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.928358 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.928339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f967d-0657-47e5-bb08-efaffbaca4fa-trusted-ca\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.929707 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.929687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef6f967d-0657-47e5-bb08-efaffbaca4fa-serving-cert\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.936454 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.936430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blb5s\" (UniqueName: \"kubernetes.io/projected/ef6f967d-0657-47e5-bb08-efaffbaca4fa-kube-api-access-blb5s\") pod \"console-operator-7866497494-hzdr6\" (UID: \"ef6f967d-0657-47e5-bb08-efaffbaca4fa\") " pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:49.984743 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:49.984558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:50.028694 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.028569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:50.029575 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.029545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv"] Jul 10 15:55:50.032390 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:50.032367 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95360d0_7c80_4f54_a510_47b017ebc24e.slice/crio-27b66f224da11497315108b0b714746246c2f78b021cea241b5e7183e047389e WatchSource:0}: Error finding container 27b66f224da11497315108b0b714746246c2f78b021cea241b5e7183e047389e: Status 404 returned error can't find the container with id 27b66f224da11497315108b0b714746246c2f78b021cea241b5e7183e047389e Jul 10 15:55:50.117893 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.117861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:50.141668 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.141573 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-hzdr6"] Jul 10 15:55:50.230242 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.230199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:50.230415 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:50.230379 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:50.230470 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:50.230458 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls podName:573e9a41-ff70-4e11-a4f4-aa94e16e6183 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:51.230441292 +0000 UTC m=+40.301259341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wk966" (UID: "573e9a41-ff70-4e11-a4f4-aa94e16e6183") : secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:50.716960 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.716920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hzdr6" event={"ID":"ef6f967d-0657-47e5-bb08-efaffbaca4fa","Type":"ContainerStarted","Data":"34c1e3456f8285e16eed0226ad02e83f359da9a762385b4bfdf41697189ea980"} Jul 10 15:55:50.718092 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:50.718059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" event={"ID":"a95360d0-7c80-4f54-a510-47b017ebc24e","Type":"ContainerStarted","Data":"27b66f224da11497315108b0b714746246c2f78b021cea241b5e7183e047389e"} Jul 10 15:55:51.128320 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.127530 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl"] Jul 10 15:55:51.145468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.145437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" Jul 10 15:55:51.150699 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.150670 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f2h5l\"" Jul 10 15:55:51.157561 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.157528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl"] Jul 10 15:55:51.238682 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.238644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:51.238886 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.238759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7f6f\" (UniqueName: \"kubernetes.io/projected/7ed9d419-3233-49fc-ba15-857098828ddf-kube-api-access-h7f6f\") pod \"network-check-source-869dcfcdcd-fjmnl\" (UID: \"7ed9d419-3233-49fc-ba15-857098828ddf\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" Jul 10 15:55:51.238886 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:51.238830 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:51.239016 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:51.238908 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls podName:573e9a41-ff70-4e11-a4f4-aa94e16e6183 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:53.238885307 +0000 UTC m=+42.309703372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wk966" (UID: "573e9a41-ff70-4e11-a4f4-aa94e16e6183") : secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:51.339495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.339459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7f6f\" (UniqueName: \"kubernetes.io/projected/7ed9d419-3233-49fc-ba15-857098828ddf-kube-api-access-h7f6f\") pod \"network-check-source-869dcfcdcd-fjmnl\" (UID: \"7ed9d419-3233-49fc-ba15-857098828ddf\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" Jul 10 15:55:51.357166 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.357132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7f6f\" (UniqueName: \"kubernetes.io/projected/7ed9d419-3233-49fc-ba15-857098828ddf-kube-api-access-h7f6f\") pod \"network-check-source-869dcfcdcd-fjmnl\" (UID: \"7ed9d419-3233-49fc-ba15-857098828ddf\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" Jul 10 15:55:51.458226 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.458185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" Jul 10 15:55:51.833994 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:51.833907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:52.072361 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:52.072318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:53.087828 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.087760 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl"] Jul 10 15:55:53.091437 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:53.091412 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9d419_3233_49fc_ba15_857098828ddf.slice/crio-b971823fab35ae24bd8030c6cda8d0c073f9dada7bb0f4a0cf8e44d579b8092f WatchSource:0}: Error finding container b971823fab35ae24bd8030c6cda8d0c073f9dada7bb0f4a0cf8e44d579b8092f: Status 404 returned error can't find the container with id b971823fab35ae24bd8030c6cda8d0c073f9dada7bb0f4a0cf8e44d579b8092f Jul 10 15:55:53.255546 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.255447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:53.255694 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:53.255609 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:53.255738 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:53.255708 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls podName:573e9a41-ff70-4e11-a4f4-aa94e16e6183 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:57.255688352 +0000 UTC m=+46.326506399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wk966" (UID: "573e9a41-ff70-4e11-a4f4-aa94e16e6183") : secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:53.558347 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.558238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:55:53.558498 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:53.558416 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 15:55:53.558498 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:53.558483 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls podName:5b65bd4f-cccb-4ae4-929d-e1f2331394a8 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:01.558467301 +0000 UTC m=+50.629285352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls") pod "dns-default-pvmrt" (UID: "5b65bd4f-cccb-4ae4-929d-e1f2331394a8") : secret "dns-default-metrics-tls" not found Jul 10 15:55:53.728499 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.728022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" event={"ID":"a95360d0-7c80-4f54-a510-47b017ebc24e","Type":"ContainerStarted","Data":"d5197545a395a71676239b4e0dd767138fe10dcfc4b8321e58a5be52bfb4850f"} Jul 10 15:55:53.729320 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.729288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" event={"ID":"7ed9d419-3233-49fc-ba15-857098828ddf","Type":"ContainerStarted","Data":"b971823fab35ae24bd8030c6cda8d0c073f9dada7bb0f4a0cf8e44d579b8092f"} Jul 10 15:55:53.731086 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.731063 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/0.log" Jul 10 15:55:53.731186 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.731104 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef6f967d-0657-47e5-bb08-efaffbaca4fa" containerID="f7c3ebb6fc4f97c73ddce98903d3c6c6c921174e9dc4d7fcea66dd7add35304b" exitCode=255 Jul 10 15:55:53.731186 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.731132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hzdr6" event={"ID":"ef6f967d-0657-47e5-bb08-efaffbaca4fa","Type":"ContainerDied","Data":"f7c3ebb6fc4f97c73ddce98903d3c6c6c921174e9dc4d7fcea66dd7add35304b"} Jul 10 15:55:53.731431 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.731413 2576 scope.go:117] "RemoveContainer" containerID="f7c3ebb6fc4f97c73ddce98903d3c6c6c921174e9dc4d7fcea66dd7add35304b" Jul 10 15:55:53.747321 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.747272 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" podStartSLOduration=1.817705173 podStartE2EDuration="4.747238561s" totalImagesPullingTime="2.929533388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:53.746474125 +0000 UTC m=+42.817292195" watchObservedRunningTime="2026-07-10 15:55:53.747238561 +0000 UTC m=+42.818056630" Jul 10 15:55:53.926969 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.926910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:53.993271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:53.992900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:54.255491 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.255457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:54.299555 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.299517 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d"] Jul 10 15:55:54.321435 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.321396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d"] Jul 10 15:55:54.321644 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.321525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" Jul 10 15:55:54.323901 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.323877 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 15:55:54.324119 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.323877 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 15:55:54.324540 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.324518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fj9n2\"" Jul 10 15:55:54.344658 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.344628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:54.413535 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.413498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:54.466037 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.465991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzszk\" (UniqueName: \"kubernetes.io/projected/2ff4b492-1992-40f8-992d-5d5d711ea24a-kube-api-access-rzszk\") pod \"migrator-7f59b54f6c-gr24d\" (UID: \"2ff4b492-1992-40f8-992d-5d5d711ea24a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" Jul 10 15:55:54.566606 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.566518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzszk\" (UniqueName: \"kubernetes.io/projected/2ff4b492-1992-40f8-992d-5d5d711ea24a-kube-api-access-rzszk\") pod \"migrator-7f59b54f6c-gr24d\" (UID: \"2ff4b492-1992-40f8-992d-5d5d711ea24a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" Jul 10 15:55:54.574686 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.574658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzszk\" (UniqueName: \"kubernetes.io/projected/2ff4b492-1992-40f8-992d-5d5d711ea24a-kube-api-access-rzszk\") pod \"migrator-7f59b54f6c-gr24d\" (UID: \"2ff4b492-1992-40f8-992d-5d5d711ea24a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" Jul 10 15:55:54.632409 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.632369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" Jul 10 15:55:54.737386 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.737353 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 15:55:54.737931 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.737905 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/0.log" Jul 10 15:55:54.738058 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.737957 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef6f967d-0657-47e5-bb08-efaffbaca4fa" containerID="6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1" exitCode=255 Jul 10 15:55:54.738119 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.738060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hzdr6" event={"ID":"ef6f967d-0657-47e5-bb08-efaffbaca4fa","Type":"ContainerDied","Data":"6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1"} Jul 10 15:55:54.738173 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.738156 2576 scope.go:117] "RemoveContainer" containerID="f7c3ebb6fc4f97c73ddce98903d3c6c6c921174e9dc4d7fcea66dd7add35304b" Jul 10 15:55:54.738323 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.738303 2576 scope.go:117] "RemoveContainer" containerID="6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1" Jul 10 15:55:54.738533 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:54.738508 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hzdr6_openshift-console-operator(ef6f967d-0657-47e5-bb08-efaffbaca4fa)\"" pod="openshift-console-operator/console-operator-7866497494-hzdr6" podUID="ef6f967d-0657-47e5-bb08-efaffbaca4fa" Jul 10 15:55:54.773882 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.773826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:54.812174 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.812139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d"] Jul 10 15:55:54.816182 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:55:54.816153 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff4b492_1992_40f8_992d_5d5d711ea24a.slice/crio-e352e5ae127f88a888394bd709afc57ddff40f702522aca4f0ad75a1d786705b WatchSource:0}: Error finding container e352e5ae127f88a888394bd709afc57ddff40f702522aca4f0ad75a1d786705b: Status 404 returned error can't find the container with id e352e5ae127f88a888394bd709afc57ddff40f702522aca4f0ad75a1d786705b Jul 10 15:55:54.870067 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:54.870022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:55.700285 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:55.700233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:55.743390 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:55.743232 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 15:55:55.743694 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:55.743672 2576 scope.go:117] "RemoveContainer" containerID="6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1" Jul 10 15:55:55.743863 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:55.743836 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hzdr6_openshift-console-operator(ef6f967d-0657-47e5-bb08-efaffbaca4fa)\"" pod="openshift-console-operator/console-operator-7866497494-hzdr6" podUID="ef6f967d-0657-47e5-bb08-efaffbaca4fa" Jul 10 15:55:55.744871 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:55.744831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" event={"ID":"2ff4b492-1992-40f8-992d-5d5d711ea24a","Type":"ContainerStarted","Data":"e352e5ae127f88a888394bd709afc57ddff40f702522aca4f0ad75a1d786705b"} Jul 10 15:55:55.765427 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:55.765383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:56.154542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.154432 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-55czr"] Jul 10 15:55:56.168371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.168334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:56.177599 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.177563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-55czr"] Jul 10 15:55:56.177773 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.177688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.180152 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.180124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 15:55:56.180674 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.180650 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 15:55:56.181468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.181128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 15:55:56.181468 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.181448 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 15:55:56.182024 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.181810 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6kc2\"" Jul 10 15:55:56.282610 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.282572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-crio-socket\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.282790 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.282637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4p9v\" (UniqueName: \"kubernetes.io/projected/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-api-access-t4p9v\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.282790 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.282686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-data-volume\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.282790 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.282717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.282790 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.282744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.312449 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.312404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:56.383118 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.383118 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.383408 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-crio-socket\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.383408 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.383232 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 15:55:56.383408 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4p9v\" (UniqueName: \"kubernetes.io/projected/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-api-access-t4p9v\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.383408 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.383361 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:56.883337707 +0000 UTC m=+45.954155807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : configmap "kube-rbac-proxy" not found Jul 10 15:55:56.383585 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-data-volume\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.383585 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383466 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-crio-socket\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.383585 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.383478 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 15:55:56.383692 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.383606 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:56.883587864 +0000 UTC m=+45.954405916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : secret "insights-runtime-extractor-tls" not found Jul 10 15:55:56.383750 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.383731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-data-volume\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.397076 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.397034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4p9v\" (UniqueName: \"kubernetes.io/projected/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-api-access-t4p9v\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.413144 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.413059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:56.507802 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.507764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:56.886950 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.886916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.887461 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:56.886963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:56.887461 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.887097 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 15:55:56.887461 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.887157 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:57.887139511 +0000 UTC m=+46.957957562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : secret "insights-runtime-extractor-tls" not found Jul 10 15:55:56.887669 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.887571 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 15:55:56.887669 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:56.887624 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:57.887607011 +0000 UTC m=+46.958425067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : configmap "kube-rbac-proxy" not found Jul 10 15:55:57.290990 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.290945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:55:57.291189 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:57.291127 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:57.291336 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:57.291214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls podName:573e9a41-ff70-4e11-a4f4-aa94e16e6183 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:05.291188399 +0000 UTC m=+54.362006453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wk966" (UID: "573e9a41-ff70-4e11-a4f4-aa94e16e6183") : secret "cluster-monitoring-operator-tls" not found Jul 10 15:55:57.753782 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.753735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" event={"ID":"2ff4b492-1992-40f8-992d-5d5d711ea24a","Type":"ContainerStarted","Data":"38afa889d80edcbaec304f52a5dbe7b2dbf75b4338fb2b8dfb9164bb40f06e42"} Jul 10 15:55:57.753966 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.753807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" event={"ID":"2ff4b492-1992-40f8-992d-5d5d711ea24a","Type":"ContainerStarted","Data":"03f7e4add547ad00a6fc87fe881daad6bf0345c1b5818f13db1b509777656adc"} Jul 10 15:55:57.755273 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.755231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" event={"ID":"7ed9d419-3233-49fc-ba15-857098828ddf","Type":"ContainerStarted","Data":"8aec3e436d29f34a011cf10febc6c20a47b5fa0e5d46472af8d5bccc403439a8"} Jul 10 15:55:57.770225 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.770172 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gr24d" podStartSLOduration=1.223368077 podStartE2EDuration="3.770156196s" totalImagesPullingTime="2.546788119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:57.769133496 +0000 UTC m=+46.839951586" watchObservedRunningTime="2026-07-10 15:55:57.770156196 +0000 UTC m=+46.840974304" Jul 10 15:55:57.783800 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.783751 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-fjmnl" podStartSLOduration=2.868075755 podStartE2EDuration="6.783736684s" totalImagesPullingTime="3.915660929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:55:57.783124088 +0000 UTC m=+46.853942182" watchObservedRunningTime="2026-07-10 15:55:57.783736684 +0000 UTC m=+46.854554753" Jul 10 15:55:57.854500 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.854402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:57.895633 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.895589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:57.895633 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.895636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:57.896113 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:57.895719 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 15:55:57.896113 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:57.895782 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 15:55:57.896113 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:57.895800 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:59.895779381 +0000 UTC m=+48.966597447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : configmap "kube-rbac-proxy" not found Jul 10 15:55:57.896113 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:57.895837 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:55:59.895821601 +0000 UTC m=+48.966639654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : secret "insights-runtime-extractor-tls" not found Jul 10 15:55:57.954554 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:57.954515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:58.041313 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:58.041278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:58.898788 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:58.898757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bwg7j_fc4cbeda-9ded-46f4-96fd-3222264a0097/dns-node-resolver/0.log" Jul 10 15:55:58.986532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:58.986493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:55:59.689364 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:59.689339 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-28vk6_97dbbccd-be38-4062-ac31-8ea70e6e6991/node-ca/0.log" Jul 10 15:55:59.908798 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:59.908754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:59.908798 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:59.908798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:55:59.909219 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:59.908889 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 15:55:59.909219 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:59.908919 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 15:55:59.909219 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:59.908974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:03.908956645 +0000 UTC m=+52.979774692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : secret "insights-runtime-extractor-tls" not found Jul 10 15:55:59.909219 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:59.908992 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:03.908982216 +0000 UTC m=+52.979800269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : configmap "kube-rbac-proxy" not found Jul 10 15:55:59.985058 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:59.984976 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:59.985194 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:59.985088 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:55:59.985461 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:55:59.985447 2576 scope.go:117] "RemoveContainer" containerID="6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1" Jul 10 15:55:59.985630 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:55:59.985611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hzdr6_openshift-console-operator(ef6f967d-0657-47e5-bb08-efaffbaca4fa)\"" pod="openshift-console-operator/console-operator-7866497494-hzdr6" podUID="ef6f967d-0657-47e5-bb08-efaffbaca4fa" Jul 10 15:56:00.494663 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:00.494631 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8z8w6_db137799-74b2-45af-9d93-b93d5939ffdb/serve-healthcheck-canary/0.log" Jul 10 15:56:00.768053 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:00.767979 2576 scope.go:117] "RemoveContainer" containerID="6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1" Jul 10 15:56:00.768186 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:00.768140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hzdr6_openshift-console-operator(ef6f967d-0657-47e5-bb08-efaffbaca4fa)\"" pod="openshift-console-operator/console-operator-7866497494-hzdr6" podUID="ef6f967d-0657-47e5-bb08-efaffbaca4fa" Jul 10 15:56:01.091239 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:01.091160 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gr24d_2ff4b492-1992-40f8-992d-5d5d711ea24a/migrator/0.log" Jul 10 15:56:01.290198 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:01.290169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gr24d_2ff4b492-1992-40f8-992d-5d5d711ea24a/graceful-termination/0.log" Jul 10 15:56:01.492958 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:01.492925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-7dtpv_a95360d0-7c80-4f54-a510-47b017ebc24e/kube-storage-version-migrator-operator/0.log" Jul 10 15:56:01.624776 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:01.624730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:56:01.624948 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:01.624866 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 15:56:01.624948 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:01.624927 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls podName:5b65bd4f-cccb-4ae4-929d-e1f2331394a8 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:17.62491342 +0000 UTC m=+66.695731468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls") pod "dns-default-pvmrt" (UID: "5b65bd4f-cccb-4ae4-929d-e1f2331394a8") : secret "dns-default-metrics-tls" not found Jul 10 15:56:02.252698 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:02.252659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:02.368106 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:02.368062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:02.551474 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:02.551371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:02.651847 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:02.651804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:02.756267 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:02.756231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:03.461305 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:03.461271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:03.779663 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:03.779581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:03.944578 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:03.944538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:03.944578 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:03.944582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:03.944841 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:03.944674 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 15:56:03.944841 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:03.944763 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:11.944746114 +0000 UTC m=+61.015564162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : configmap "kube-rbac-proxy" not found Jul 10 15:56:03.944841 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:03.944767 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 15:56:03.944841 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:03.944820 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:11.944803743 +0000 UTC m=+61.015621809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : secret "insights-runtime-extractor-tls" not found Jul 10 15:56:04.447335 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:04.447295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:04.592219 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:04.592182 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:04.967623 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:04.967593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:05.357424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:05.357326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:56:05.357581 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:05.357486 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 15:56:05.357581 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:05.357561 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls podName:573e9a41-ff70-4e11-a4f4-aa94e16e6183 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:21.357544173 +0000 UTC m=+70.428362220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wk966" (UID: "573e9a41-ff70-4e11-a4f4-aa94e16e6183") : secret "cluster-monitoring-operator-tls" not found Jul 10 15:56:05.373589 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:05.373560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:05.755072 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:05.755037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:05.952169 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:05.952135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:06.189369 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:06.189337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:06.290908 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:06.290857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:06.555272 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:06.555156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:06.953195 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:06.953159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:07.012628 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:07.012587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:07.403135 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:07.403050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:07.501470 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:07.501441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:07.594014 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:07.593981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:08.164801 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:08.164760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:08.269079 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:08.269047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:08.554607 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:08.554554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:08.766101 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:08.766063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:09.152355 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:09.152311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:09.980220 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:09.980180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:10.345425 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:10.345304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:10.418612 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:10.418569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:10.564750 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:10.564718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:10.761668 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:10.761632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:10.961990 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:10.961957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:11.690385 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:11.690359 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssxhb" Jul 10 15:56:11.953338 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:11.953230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:12.008626 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.008594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:12.008626 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.008630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:12.008825 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:12.008747 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 15:56:12.008825 ip-10-0-139-69 kubenswrapper[2576]: E0710 15:56:12.008801 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls podName:5ccde3d4-bbf9-44ac-a89c-f36b83af6db2 nodeName:}" failed. No retries permitted until 2026-07-10 15:56:28.008785535 +0000 UTC m=+77.079603583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55czr" (UID: "5ccde3d4-bbf9-44ac-a89c-f36b83af6db2") : secret "insights-runtime-extractor-tls" not found Jul 10 15:56:12.009174 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.009154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:12.358657 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.358565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:12.561605 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.561545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:12.755403 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.755368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:12.953669 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:12.953635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:14.489895 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.489859 2576 scope.go:117] "RemoveContainer" containerID="6e753f15a5bac74ba0f3aa214eccc4b63f12c827cd8fc5e9a959c2054f7695c1" Jul 10 15:56:14.811215 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.811139 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 15:56:14.811369 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.811221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hzdr6" event={"ID":"ef6f967d-0657-47e5-bb08-efaffbaca4fa","Type":"ContainerStarted","Data":"79ec0c0dbacbdf0078d7c838a0062415b53f141e51e31f6e4ec6c933915ccdef"} Jul 10 15:56:14.811635 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.811614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:56:14.830186 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.830133 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-hzdr6" podStartSLOduration=23.011399641 podStartE2EDuration="25.830117543s" totalImagesPullingTime="2.818717902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:14.829651753 +0000 UTC m=+63.900469822" watchObservedRunningTime="2026-07-10 15:56:14.830117543 +0000 UTC m=+63.900935612" Jul 10 15:56:14.851614 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.851581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:14.915031 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:14.914996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:15.249631 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:15.249602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-hzdr6" Jul 10 15:56:15.808449 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:15.808416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:16.142974 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.142881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:56:16.142974 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.142961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:56:16.145387 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.145365 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 15:56:16.145572 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.145555 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksw9\" (UniqueName: \"kubernetes.io/projected/e7076633-c37e-4dca-881c-1501718695db-kube-api-access-rksw9\") pod \"network-check-target-ddv99\" (UID: \"e7076633-c37e-4dca-881c-1501718695db\") " pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:56:16.156267 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.156236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58620b37-9b5a-40fb-b71b-d2c06e229147-original-pull-secret\") pod \"global-pull-secret-syncer-kw467\" (UID: \"58620b37-9b5a-40fb-b71b-d2c06e229147\") " pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:56:16.199565 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.199524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw467" Jul 10 15:56:16.224314 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.224281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:16.243417 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.243382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:56:16.246146 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.246122 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 15:56:16.256505 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.256475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dae35b6-cf5b-4c24-9390-da5f1e0d93fc-metrics-certs\") pod \"network-metrics-daemon-85zw9\" (UID: \"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc\") " pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:56:16.294372 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.294232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:16.304853 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.304825 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2tf4w\"" Jul 10 15:56:16.309547 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.309525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8rkdc\"" Jul 10 15:56:16.313186 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.313163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:56:16.318104 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.318088 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85zw9" Jul 10 15:56:16.326392 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.326324 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kw467"] Jul 10 15:56:16.330122 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:16.330096 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58620b37_9b5a_40fb_b71b_d2c06e229147.slice/crio-c235a6ea18592cec9421d9d37960e62da852c892045b04255ab6e0ecf08ca1f6 WatchSource:0}: Error finding container c235a6ea18592cec9421d9d37960e62da852c892045b04255ab6e0ecf08ca1f6: Status 404 returned error can't find the container with id c235a6ea18592cec9421d9d37960e62da852c892045b04255ab6e0ecf08ca1f6 Jul 10 15:56:16.442740 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.442707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddv99"] Jul 10 15:56:16.446419 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:16.446386 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7076633_c37e_4dca_881c_1501718695db.slice/crio-9a4f16268cf8657920fa921ca06227c807667b5dc4eeae34e09fc038f5953569 WatchSource:0}: Error finding container 9a4f16268cf8657920fa921ca06227c807667b5dc4eeae34e09fc038f5953569: Status 404 returned error can't find the container with id 9a4f16268cf8657920fa921ca06227c807667b5dc4eeae34e09fc038f5953569 Jul 10 15:56:16.459420 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.459397 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85zw9"] Jul 10 15:56:16.462390 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:16.462365 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dae35b6_cf5b_4c24_9390_da5f1e0d93fc.slice/crio-29864b979ebfd35c5ff234425e5e0cdfc427232592ea8cfa53b408f8e1692b20 WatchSource:0}: Error finding container 29864b979ebfd35c5ff234425e5e0cdfc427232592ea8cfa53b408f8e1692b20: Status 404 returned error can't find the container with id 29864b979ebfd35c5ff234425e5e0cdfc427232592ea8cfa53b408f8e1692b20 Jul 10 15:56:16.509240 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.509201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:16.819138 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.819102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85zw9" event={"ID":"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc","Type":"ContainerStarted","Data":"29864b979ebfd35c5ff234425e5e0cdfc427232592ea8cfa53b408f8e1692b20"} Jul 10 15:56:16.821047 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.821017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddv99" event={"ID":"e7076633-c37e-4dca-881c-1501718695db","Type":"ContainerStarted","Data":"9850b7842390de2d57a6b9ebe958075f288671cd7a847cd23eb3d8a855f0fb91"} Jul 10 15:56:16.821186 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.821064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddv99" event={"ID":"e7076633-c37e-4dca-881c-1501718695db","Type":"ContainerStarted","Data":"9a4f16268cf8657920fa921ca06227c807667b5dc4eeae34e09fc038f5953569"} Jul 10 15:56:16.821186 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.821105 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:56:16.822274 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.822231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kw467" event={"ID":"58620b37-9b5a-40fb-b71b-d2c06e229147","Type":"ContainerStarted","Data":"c235a6ea18592cec9421d9d37960e62da852c892045b04255ab6e0ecf08ca1f6"} Jul 10 15:56:16.837636 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.837578 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ddv99" podStartSLOduration=65.837561456 podStartE2EDuration="1m5.837561456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:16.836969597 +0000 UTC m=+65.907787673" watchObservedRunningTime="2026-07-10 15:56:16.837561456 +0000 UTC m=+65.908379526" Jul 10 15:56:16.921904 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:16.921867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:17.158047 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.157962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:17.222351 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.222309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:17.326935 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.326875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:17.621021 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.620981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:17.653545 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.653502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:56:17.656506 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.656479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b65bd4f-cccb-4ae4-929d-e1f2331394a8-metrics-tls\") pod \"dns-default-pvmrt\" (UID: \"5b65bd4f-cccb-4ae4-929d-e1f2331394a8\") " pod="openshift-dns/dns-default-pvmrt" Jul 10 15:56:17.931657 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.931623 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nh8x5\"" Jul 10 15:56:17.939857 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:17.939825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvmrt" Jul 10 15:56:18.093717 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:18.093687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvmrt"] Jul 10 15:56:18.097052 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:18.097024 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b65bd4f_cccb_4ae4_929d_e1f2331394a8.slice/crio-431bf043019b78a87016539c3b539b48453a97fecb23f84017dd5344257a2aa8 WatchSource:0}: Error finding container 431bf043019b78a87016539c3b539b48453a97fecb23f84017dd5344257a2aa8: Status 404 returned error can't find the container with id 431bf043019b78a87016539c3b539b48453a97fecb23f84017dd5344257a2aa8 Jul 10 15:56:18.831233 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:18.831194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvmrt" event={"ID":"5b65bd4f-cccb-4ae4-929d-e1f2331394a8","Type":"ContainerStarted","Data":"431bf043019b78a87016539c3b539b48453a97fecb23f84017dd5344257a2aa8"} Jul 10 15:56:18.833334 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:18.833307 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85zw9" event={"ID":"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc","Type":"ContainerStarted","Data":"dd1847687413e7b7b4245056184b387e26133587faba0270811bed937acd1abc"} Jul 10 15:56:18.833463 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:18.833361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85zw9" event={"ID":"3dae35b6-cf5b-4c24-9390-da5f1e0d93fc","Type":"ContainerStarted","Data":"1a408a3c29bb0bb703a9b876a1053683d3819c66369df0c60ef88bace13a5b97"} Jul 10 15:56:18.852152 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:18.852097 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-85zw9" podStartSLOduration=66.514966001 podStartE2EDuration="1m7.852077223s" totalImagesPullingTime="1.337111222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:18.851237844 +0000 UTC m=+67.922055913" watchObservedRunningTime="2026-07-10 15:56:18.852077223 +0000 UTC m=+67.922895293" Jul 10 15:56:19.354443 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:19.354397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:21.382046 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.381962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:56:21.384566 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.384545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/573e9a41-ff70-4e11-a4f4-aa94e16e6183-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wk966\" (UID: \"573e9a41-ff70-4e11-a4f4-aa94e16e6183\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:56:21.576626 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.576594 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7rqsb\"" Jul 10 15:56:21.584152 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.584117 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" Jul 10 15:56:21.727089 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.727057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966"] Jul 10 15:56:21.730184 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:21.730149 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573e9a41_ff70_4e11_a4f4_aa94e16e6183.slice/crio-23378a78ff2cac81c54a4c0b756cc1c6344df15d1ffebb0e82b9ea2d743ffc4f WatchSource:0}: Error finding container 23378a78ff2cac81c54a4c0b756cc1c6344df15d1ffebb0e82b9ea2d743ffc4f: Status 404 returned error can't find the container with id 23378a78ff2cac81c54a4c0b756cc1c6344df15d1ffebb0e82b9ea2d743ffc4f Jul 10 15:56:21.845201 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.845164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvmrt" event={"ID":"5b65bd4f-cccb-4ae4-929d-e1f2331394a8","Type":"ContainerStarted","Data":"9dbe35e97b6d3f8a267d4d70f640b90b6cd371b4de3f90bf4f5ba89371382f3f"} Jul 10 15:56:21.845395 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.845223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvmrt" event={"ID":"5b65bd4f-cccb-4ae4-929d-e1f2331394a8","Type":"ContainerStarted","Data":"649823dc2b519638bcedaaf7f1942bea57c04ef01618c9a409015f2624b7c1f7"} Jul 10 15:56:21.845395 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.845288 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvmrt" Jul 10 15:56:21.846843 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.846817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kw467" event={"ID":"58620b37-9b5a-40fb-b71b-d2c06e229147","Type":"ContainerStarted","Data":"a2b2ad854a305a8d99cae44520eeaeb15d8b6831c373f84db2e7f34499d9397e"} Jul 10 15:56:21.847940 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.847920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" event={"ID":"573e9a41-ff70-4e11-a4f4-aa94e16e6183","Type":"ContainerStarted","Data":"23378a78ff2cac81c54a4c0b756cc1c6344df15d1ffebb0e82b9ea2d743ffc4f"} Jul 10 15:56:21.862814 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.862761 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pvmrt" podStartSLOduration=33.909496276 podStartE2EDuration="36.862744274s" totalImagesPullingTime="2.953247998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:21.861863054 +0000 UTC m=+70.932681125" watchObservedRunningTime="2026-07-10 15:56:21.862744274 +0000 UTC m=+70.933562349" Jul 10 15:56:21.878038 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:21.877986 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kw467" podStartSLOduration=66.160529867 podStartE2EDuration="1m10.877970842s" totalImagesPullingTime="4.717440975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:21.87708742 +0000 UTC m=+70.947905489" watchObservedRunningTime="2026-07-10 15:56:21.877970842 +0000 UTC m=+70.948788911" Jul 10 15:56:22.312308 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.311922 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v"] Jul 10 15:56:22.314443 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.314418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.323352 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.323155 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 15:56:22.323352 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.323155 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 15:56:22.323352 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.323158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 15:56:22.323784 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.323765 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 15:56:22.323919 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.323768 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 15:56:22.325158 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.325136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 15:56:22.325495 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.325479 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 15:56:22.359460 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.359427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v"] Jul 10 15:56:22.391274 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.391225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1434126-b21e-48b4-8dc0-381b45cd3acd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.391679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.391310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.391679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.391339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd2n\" (UniqueName: \"kubernetes.io/projected/b1434126-b21e-48b4-8dc0-381b45cd3acd-kube-api-access-mkd2n\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.391679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.391378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-ca\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.391679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.391439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-hub\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.391679 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.391467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.492227 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.492186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1434126-b21e-48b4-8dc0-381b45cd3acd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.492413 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.492238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.492413 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.492283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd2n\" (UniqueName: \"kubernetes.io/projected/b1434126-b21e-48b4-8dc0-381b45cd3acd-kube-api-access-mkd2n\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.492413 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.492308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-ca\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.492413 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.492335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-hub\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.492413 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.492350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.493423 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.493395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1434126-b21e-48b4-8dc0-381b45cd3acd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.495693 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.495668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.495803 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.495724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.495865 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.495821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-hub\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.495932 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.495875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b1434126-b21e-48b4-8dc0-381b45cd3acd-ca\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.505385 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.505362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd2n\" (UniqueName: \"kubernetes.io/projected/b1434126-b21e-48b4-8dc0-381b45cd3acd-kube-api-access-mkd2n\") pod \"cluster-proxy-proxy-agent-759b76dd9f-v6k7v\" (UID: \"b1434126-b21e-48b4-8dc0-381b45cd3acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.620040 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.619931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:22.640507 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.640472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" Jul 10 15:56:22.716111 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.716027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:22.804925 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.804891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v"] Jul 10 15:56:22.816272 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.816219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:22.981730 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:22.981694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:23.069552 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:23.069517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:23.270556 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:23.270476 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1434126_b21e_48b4_8dc0_381b45cd3acd.slice/crio-2c0336352d763756e027ff62713ff15efda98932bed83066f1877ba2c9c364c3 WatchSource:0}: Error finding container 2c0336352d763756e027ff62713ff15efda98932bed83066f1877ba2c9c364c3: Status 404 returned error can't find the container with id 2c0336352d763756e027ff62713ff15efda98932bed83066f1877ba2c9c364c3 Jul 10 15:56:23.719509 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:23.719468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:23.817532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:23.817497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:23.856081 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:23.856043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" event={"ID":"b1434126-b21e-48b4-8dc0-381b45cd3acd","Type":"ContainerStarted","Data":"2c0336352d763756e027ff62713ff15efda98932bed83066f1877ba2c9c364c3"} Jul 10 15:56:23.857617 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:23.857591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" event={"ID":"573e9a41-ff70-4e11-a4f4-aa94e16e6183","Type":"ContainerStarted","Data":"8265ef047a451575af2d467d801b7cc7bcb3a654713cdec2ad6253cdf7e4fcea"} Jul 10 15:56:23.943574 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:23.943537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:24.064955 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:24.064862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:26.755641 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:26.755583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:26.830513 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:26.830479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:27.873916 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:27.873871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" event={"ID":"b1434126-b21e-48b4-8dc0-381b45cd3acd","Type":"ContainerStarted","Data":"77e6fac5d18d8e27f3bf2623ca5352e5918add019ae0b03de417033f4a6d6c03"} Jul 10 15:56:27.874323 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:27.873923 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" event={"ID":"b1434126-b21e-48b4-8dc0-381b45cd3acd","Type":"ContainerStarted","Data":"4c40e96341d4538234b31d3e1736391947fcf6a74e4a6b4dc5db509738aa2fd2"} Jul 10 15:56:27.874323 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:27.873942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" event={"ID":"b1434126-b21e-48b4-8dc0-381b45cd3acd","Type":"ContainerStarted","Data":"92a521f35dd2eaffc5e7a2620747a7a79edddc8d8130365ee21374417ec0c858"} Jul 10 15:56:27.900532 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:27.900476 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-759b76dd9f-v6k7v" podStartSLOduration=2.323027013 podStartE2EDuration="5.900457156s" totalImagesPullingTime="3.577430143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:27.900236205 +0000 UTC m=+76.971054297" watchObservedRunningTime="2026-07-10 15:56:27.900457156 +0000 UTC m=+76.971275226" Jul 10 15:56:27.900770 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:27.900742 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wk966" podStartSLOduration=37.33728213 podStartE2EDuration="38.900734456s" totalImagesPullingTime="1.563452326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:23.894350261 +0000 UTC m=+72.965168330" watchObservedRunningTime="2026-07-10 15:56:27.900734456 +0000 UTC m=+76.971552879" Jul 10 15:56:28.037831 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.037794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:28.040298 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.040277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ccde3d4-bbf9-44ac-a89c-f36b83af6db2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55czr\" (UID: \"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2\") " pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:28.295309 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.295278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6kc2\"" Jul 10 15:56:28.302983 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.302964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-55czr" Jul 10 15:56:28.456860 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.456829 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-55czr"] Jul 10 15:56:28.460008 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:28.459981 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccde3d4_bbf9_44ac_a89c_f36b83af6db2.slice/crio-ac9ae4739e22234fd8100abba19ef70bd7a7fb5423d388c5870502b52f9fcb90 WatchSource:0}: Error finding container ac9ae4739e22234fd8100abba19ef70bd7a7fb5423d388c5870502b52f9fcb90: Status 404 returned error can't find the container with id ac9ae4739e22234fd8100abba19ef70bd7a7fb5423d388c5870502b52f9fcb90 Jul 10 15:56:28.878077 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.878041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55czr" event={"ID":"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2","Type":"ContainerStarted","Data":"ac9ae4739e22234fd8100abba19ef70bd7a7fb5423d388c5870502b52f9fcb90"} Jul 10 15:56:28.935763 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:28.935717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:29.008047 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:29.007956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:29.999484 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:29.999447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:30.065375 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:30.065238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:30.131279 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:30.131217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:30.634148 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:30.634104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:30.735362 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:30.735325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:30.808687 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:30.808656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:31.751136 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:31.750990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:31.813021 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:31.812973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:31.854575 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:31.854541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvmrt" Jul 10 15:56:31.873909 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:31.873873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:31.979564 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:31.979524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:32.505137 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:32.505050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:32.606334 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:32.606292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:32.894714 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:32.894629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55czr" event={"ID":"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2","Type":"ContainerStarted","Data":"677dddd1caa0f6981ac137abca56df84f04a156f51f001b8a0bbbc5eb9d5f572"} Jul 10 15:56:33.899450 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:33.899419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55czr" event={"ID":"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2","Type":"ContainerStarted","Data":"bf71b3e4369ed65a001cb589c7e6de5cd7a601638afb24c8ed0004407cb3e863"} Jul 10 15:56:34.821893 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:34.821859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:34.958741 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:34.958707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:35.065050 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:35.065009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:36.912481 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:36.912440 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55czr" event={"ID":"5ccde3d4-bbf9-44ac-a89c-f36b83af6db2","Type":"ContainerStarted","Data":"b54b6c868fdba545ce11c908268a076bc95ab3fe30caf402bcc2a0e048d6f427"} Jul 10 15:56:36.932078 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:36.932020 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-55czr" podStartSLOduration=33.289418535 podStartE2EDuration="40.932004161s" totalImagesPullingTime="7.642585626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:55:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:36.930537633 +0000 UTC m=+86.001355713" watchObservedRunningTime="2026-07-10 15:56:36.932004161 +0000 UTC m=+86.002822231" Jul 10 15:56:37.137105 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.137070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:37.396108 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.396066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:37.434772 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.434739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx"] Jul 10 15:56:37.438036 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.438017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.440632 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.440609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 15:56:37.440760 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.440612 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7cbps\"" Jul 10 15:56:37.440760 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.440668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 15:56:37.441108 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.441095 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 15:56:37.451319 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.451298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx"] Jul 10 15:56:37.468984 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.468950 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ztlh4"] Jul 10 15:56:37.472134 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.472116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.474540 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.474510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nn7fg\"" Jul 10 15:56:37.474540 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.474534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 15:56:37.474846 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.474828 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 15:56:37.475071 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.475056 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 15:56:37.511485 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.511453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.511645 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.511508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.511645 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.511538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4nw\" (UniqueName: \"kubernetes.io/projected/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-kube-api-access-9n4nw\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.511645 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.511560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.593195 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.593163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:37.612195 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.612371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4nw\" (UniqueName: \"kubernetes.io/projected/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-kube-api-access-9n4nw\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.612371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.612371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqj8v\" (UniqueName: \"kubernetes.io/projected/fc9abbc1-7d87-453f-a05f-f3147681ca3f-kube-api-access-jqj8v\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-tls\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612371 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-wtmp\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612667 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.612667 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612667 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-root\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612667 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-textfile\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612667 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-sys\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.612667 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.612597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc9abbc1-7d87-453f-a05f-f3147681ca3f-metrics-client-ca\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.613314 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.613291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.614818 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.614798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.614818 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.614805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.637144 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.637115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4nw\" (UniqueName: \"kubernetes.io/projected/d9afae43-1a88-4555-a8e2-7d4148f0f0d8-kube-api-access-9n4nw\") pod \"openshift-state-metrics-84bd68b896-vhhfx\" (UID: \"d9afae43-1a88-4555-a8e2-7d4148f0f0d8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.713211 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqj8v\" (UniqueName: \"kubernetes.io/projected/fc9abbc1-7d87-453f-a05f-f3147681ca3f-kube-api-access-jqj8v\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713211 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-tls\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713449 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-wtmp\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713449 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713518 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-root\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713518 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-textfile\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713518 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-sys\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713639 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-root\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713639 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc9abbc1-7d87-453f-a05f-f3147681ca3f-metrics-client-ca\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713639 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713759 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-sys\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713759 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-wtmp\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.713874 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.713855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-textfile\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.714102 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.714083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.714379 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.714246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc9abbc1-7d87-453f-a05f-f3147681ca3f-metrics-client-ca\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.716121 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.716095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.716228 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.716095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc9abbc1-7d87-453f-a05f-f3147681ca3f-node-exporter-tls\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.725969 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.725943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:37.736692 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.736671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqj8v\" (UniqueName: \"kubernetes.io/projected/fc9abbc1-7d87-453f-a05f-f3147681ca3f-kube-api-access-jqj8v\") pod \"node-exporter-ztlh4\" (UID: \"fc9abbc1-7d87-453f-a05f-f3147681ca3f\") " pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.747605 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.747584 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" Jul 10 15:56:37.781975 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.781937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ztlh4" Jul 10 15:56:37.794059 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:37.794024 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9abbc1_7d87_453f_a05f_f3147681ca3f.slice/crio-6acba5f768dc361de93fd5f2e1ac1556aeeacc0d941f678bc5f26adb71887ac4 WatchSource:0}: Error finding container 6acba5f768dc361de93fd5f2e1ac1556aeeacc0d941f678bc5f26adb71887ac4: Status 404 returned error can't find the container with id 6acba5f768dc361de93fd5f2e1ac1556aeeacc0d941f678bc5f26adb71887ac4 Jul 10 15:56:37.820134 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.820101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:37.887312 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.887287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx"] Jul 10 15:56:37.889592 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:37.889566 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9afae43_1a88_4555_a8e2_7d4148f0f0d8.slice/crio-d8000b65a5616a4f9484624b3fbee98109039f4e528ae887ab7162697fc3c4de WatchSource:0}: Error finding container d8000b65a5616a4f9484624b3fbee98109039f4e528ae887ab7162697fc3c4de: Status 404 returned error can't find the container with id d8000b65a5616a4f9484624b3fbee98109039f4e528ae887ab7162697fc3c4de Jul 10 15:56:37.916816 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.916784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ztlh4" event={"ID":"fc9abbc1-7d87-453f-a05f-f3147681ca3f","Type":"ContainerStarted","Data":"6acba5f768dc361de93fd5f2e1ac1556aeeacc0d941f678bc5f26adb71887ac4"} Jul 10 15:56:37.917929 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.917905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" event={"ID":"d9afae43-1a88-4555-a8e2-7d4148f0f0d8","Type":"ContainerStarted","Data":"d8000b65a5616a4f9484624b3fbee98109039f4e528ae887ab7162697fc3c4de"} Jul 10 15:56:37.931405 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:37.931364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:38.219302 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.219196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:38.371069 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.371033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:38.479408 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.479313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:38.696958 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.696910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:38.798058 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.797962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:38.925430 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.925389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" event={"ID":"d9afae43-1a88-4555-a8e2-7d4148f0f0d8","Type":"ContainerStarted","Data":"9a2c6ac76b5c0fedb970a26a26007f77f49a03733cb4f1bd3a50e5ca264ce1b9"} Jul 10 15:56:38.925878 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:38.925463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" event={"ID":"d9afae43-1a88-4555-a8e2-7d4148f0f0d8","Type":"ContainerStarted","Data":"5d603d4fff48595508e0eb1c217430141656001591e77e3195ad06af12a53911"} Jul 10 15:56:39.930924 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:39.930880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" event={"ID":"d9afae43-1a88-4555-a8e2-7d4148f0f0d8","Type":"ContainerStarted","Data":"6beee1698f4089c59eba55e7d31c17739aab17fa56cd0dc6d5e132c497ddba89"} Jul 10 15:56:39.932584 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:39.932562 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc9abbc1-7d87-453f-a05f-f3147681ca3f" containerID="e4b5500c418fae446022189b99dd37f8a1fca61a8fbfce2f8b1dc6a88132e859" exitCode=0 Jul 10 15:56:39.932641 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:39.932603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ztlh4" event={"ID":"fc9abbc1-7d87-453f-a05f-f3147681ca3f","Type":"ContainerDied","Data":"e4b5500c418fae446022189b99dd37f8a1fca61a8fbfce2f8b1dc6a88132e859"} Jul 10 15:56:39.951235 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:39.951173 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vhhfx" podStartSLOduration=1.8667168520000001 podStartE2EDuration="2.951152024s" totalImagesPullingTime="1.084435172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:39.950443725 +0000 UTC m=+89.021261822" watchObservedRunningTime="2026-07-10 15:56:39.951152024 +0000 UTC m=+89.021970094" Jul 10 15:56:40.153025 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.152984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:40.415375 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.415338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:40.491202 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.491157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-58b7fb7f8f-f927w"] Jul 10 15:56:40.494813 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.494795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.497465 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.497441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 15:56:40.497925 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.497904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 15:56:40.498041 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.497935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 15:56:40.498041 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.497935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 15:56:40.498041 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.497983 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bd2g330k8i9l\"" Jul 10 15:56:40.498227 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.498194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 15:56:40.499861 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.499842 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b5q55\"" Jul 10 15:56:40.512314 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.512287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-58b7fb7f8f-f927w"] Jul 10 15:56:40.516399 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.516373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:40.616618 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.616582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:40.637622 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637713 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/82924150-518a-4279-91a3-04dc86dd1785-kube-api-access-4bm59\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637713 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-tls\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637713 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-grpc-tls\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637833 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637833 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637833 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.637833 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.637815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82924150-518a-4279-91a3-04dc86dd1785-metrics-client-ca\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.716356 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.716315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:40.738367 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738367 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738563 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738563 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82924150-518a-4279-91a3-04dc86dd1785-metrics-client-ca\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738563 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738563 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/82924150-518a-4279-91a3-04dc86dd1785-kube-api-access-4bm59\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738563 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-tls\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.738563 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.738550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-grpc-tls\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.739391 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.739360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82924150-518a-4279-91a3-04dc86dd1785-metrics-client-ca\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.741271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.741106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.741271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.741184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.741590 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.741559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.741702 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.741650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.741913 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.741887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-thanos-querier-tls\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.741986 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.741963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82924150-518a-4279-91a3-04dc86dd1785-secret-grpc-tls\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.754710 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.754685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/82924150-518a-4279-91a3-04dc86dd1785-kube-api-access-4bm59\") pod \"thanos-querier-58b7fb7f8f-f927w\" (UID: \"82924150-518a-4279-91a3-04dc86dd1785\") " pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.803885 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.803842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:40.938674 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.938634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ztlh4" event={"ID":"fc9abbc1-7d87-453f-a05f-f3147681ca3f","Type":"ContainerStarted","Data":"d9e8d36dbe88e02ae9c40450f0afcf585fc672fad7c8301b78ada38704fa2527"} Jul 10 15:56:40.939032 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.938700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ztlh4" event={"ID":"fc9abbc1-7d87-453f-a05f-f3147681ca3f","Type":"ContainerStarted","Data":"1faedce875184b30633709a3a17c7b5377eea4a4b6fb5fbc5e4d3174cb264d5b"} Jul 10 15:56:40.954643 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.954613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-58b7fb7f8f-f927w"] Jul 10 15:56:40.957388 ip-10-0-139-69 kubenswrapper[2576]: W0710 15:56:40.957356 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82924150_518a_4279_91a3_04dc86dd1785.slice/crio-855fda9a0b0944b2cb132de374d96853c8a6942efb34aa8dd360d7a39b182e2e WatchSource:0}: Error finding container 855fda9a0b0944b2cb132de374d96853c8a6942efb34aa8dd360d7a39b182e2e: Status 404 returned error can't find the container with id 855fda9a0b0944b2cb132de374d96853c8a6942efb34aa8dd360d7a39b182e2e Jul 10 15:56:40.959384 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:40.959336 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ztlh4" podStartSLOduration=2.569906863 podStartE2EDuration="3.959321266s" totalImagesPullingTime="1.312684787s" totalInitContainerRuntime="76.729616ms" isStatefulPod=true podCreationTimestamp="2026-07-10 15:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:40.958066925 +0000 UTC m=+90.028884999" watchObservedRunningTime="2026-07-10 15:56:40.959321266 +0000 UTC m=+90.030139336" Jul 10 15:56:41.071005 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:41.070910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:41.172221 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:41.172179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:41.524188 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:41.524143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:41.610322 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:41.610278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:41.944271 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:41.944224 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"855fda9a0b0944b2cb132de374d96853c8a6942efb34aa8dd360d7a39b182e2e"} Jul 10 15:56:42.191151 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:42.191110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:42.252886 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:42.252791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:42.498777 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:42.498740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:43.353564 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.353529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:43.454424 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.454373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:43.719751 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.719710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:43.959269 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.959221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"8507dae78a4808a2fbae6320daa90167f5bf291a8c67f610c475f34f8ce88a05"} Jul 10 15:56:43.959442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.959309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"718bcb7f6ca51a14909ad3e51c9873224a5e49cdeb9cce0cfc3997bce01431b7"} Jul 10 15:56:43.959442 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.959337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"c37ddf432da3e1235c0ce5f4dda2ad24966314da0d855df2b3db32ce303d064e"} Jul 10 15:56:43.972764 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:43.972736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:44.071655 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.071619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:44.850315 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.850271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:44.950740 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.950705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:44.965381 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.965345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"3f108e38b767c053f7149bf970ef59f31b2c307b01ca8596f26fd73adcad4847"} Jul 10 15:56:44.965564 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.965443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"d838a063637be3e2f3034f21e53ba963aac351786d12eded0bf29a20dc627ecb"} Jul 10 15:56:44.965564 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.965478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" event={"ID":"82924150-518a-4279-91a3-04dc86dd1785","Type":"ContainerStarted","Data":"0f2f36981e3ecee8bb5b76373e59d5a0419da15d301a4570956251366f7dd803"} Jul 10 15:56:44.965564 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.965507 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:44.991520 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:44.991469 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" podStartSLOduration=1.531547443 podStartE2EDuration="4.991452561s" totalImagesPullingTime="3.459905118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:56:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:56:44.990312649 +0000 UTC m=+94.061130720" watchObservedRunningTime="2026-07-10 15:56:44.991452561 +0000 UTC m=+94.062270634" Jul 10 15:56:47.568852 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:47.568819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:47.631670 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:47.631630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:47.828542 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:47.828465 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ddv99" Jul 10 15:56:47.941351 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:47.941314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:48.006590 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:48.006557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:48.066373 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:48.066327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:48.166951 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:48.166861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:50.512772 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:50.512735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:50.975895 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:50.975865 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-58b7fb7f8f-f927w" Jul 10 15:56:52.780067 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:52.780028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:52.879937 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:52.879906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:52.935716 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:52.935684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:53.215303 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:53.215243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:53.311313 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:53.311278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:59.470080 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:59.470040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:59.852582 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:59.852490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:56:59.918715 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:56:59.918678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:04.036057 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:04.036024 2576 generic.go:356] "Generic (PLEG): container finished" podID="a95360d0-7c80-4f54-a510-47b017ebc24e" containerID="d5197545a395a71676239b4e0dd767138fe10dcfc4b8321e58a5be52bfb4850f" exitCode=0 Jul 10 15:57:04.036409 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:04.036099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" event={"ID":"a95360d0-7c80-4f54-a510-47b017ebc24e","Type":"ContainerDied","Data":"d5197545a395a71676239b4e0dd767138fe10dcfc4b8321e58a5be52bfb4850f"} Jul 10 15:57:04.036455 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:04.036414 2576 scope.go:117] "RemoveContainer" containerID="d5197545a395a71676239b4e0dd767138fe10dcfc4b8321e58a5be52bfb4850f" Jul 10 15:57:05.041822 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:05.041782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7dtpv" event={"ID":"a95360d0-7c80-4f54-a510-47b017ebc24e","Type":"ContainerStarted","Data":"b42d1b8c446c7bc23857767da6871a516b73ac08ba9afed0fea654db264b2780"} Jul 10 15:57:07.327064 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:07.327026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:07.427227 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:07.427194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:07.492572 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:07.492538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:09.301303 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:09.301223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:13.823192 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:13.823158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:13.886579 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:13.886541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:14.868014 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:14.867974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:14.931892 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:14.931851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:53.068497 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:53.068464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:56.738311 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:56.738246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:57:56.960827 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:57:56.960789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:05.708548 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:05.708454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:05.855150 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:05.855117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:05.955409 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:05.955368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:06.499777 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:06.499741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:06.566615 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:06.566579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:12.035222 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:12.035178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:12.101005 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:12.100955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:24.075491 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:24.075451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:24.145525 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:24.145492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:26.179143 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:26.179110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:26.380162 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:26.380109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:26.512353 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:26.512311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:29.862384 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:29.862350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:29.962155 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:29.962098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:47.139001 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:47.138967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:58:47.240008 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:58:47.239970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:08.383466 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:08.383403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:08.484537 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:08.484500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:35.211149 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:35.211064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:36.217990 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:36.217945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:38.823159 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:38.823119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:38.921618 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:38.921580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:43.606153 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:43.606118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:43.666071 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:43.666023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:52.792547 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:52.792493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:52.897800 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:52.897758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:55.170391 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:55.170353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:55.618521 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:55.618482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:56.772097 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:56.772065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 15:59:56.835306 ip-10-0-139-69 kubenswrapper[2576]: I0710 15:59:56.835265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:00.676166 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:00.676132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:00.777352 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:00.777306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:02.525888 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:02.525860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:03.173698 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.173666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:03.267071 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.267036 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-chz7z"] Jul 10 16:00:03.269194 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.269176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.271242 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.271216 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 16:00:03.271982 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.271964 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-b45df\"" Jul 10 16:00:03.272092 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.271973 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 16:00:03.288351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.288326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-chz7z"] Jul 10 16:00:03.312182 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.312147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:03.453480 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.453444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffd55d6b-1688-46f3-b2e1-7c68f86b908d-bound-sa-token\") pod \"cert-manager-55b4d954d5-chz7z\" (UID: \"ffd55d6b-1688-46f3-b2e1-7c68f86b908d\") " pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.453656 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.453514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66th\" (UniqueName: \"kubernetes.io/projected/ffd55d6b-1688-46f3-b2e1-7c68f86b908d-kube-api-access-d66th\") pod \"cert-manager-55b4d954d5-chz7z\" (UID: \"ffd55d6b-1688-46f3-b2e1-7c68f86b908d\") " pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.554037 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.553999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66th\" (UniqueName: \"kubernetes.io/projected/ffd55d6b-1688-46f3-b2e1-7c68f86b908d-kube-api-access-d66th\") pod \"cert-manager-55b4d954d5-chz7z\" (UID: \"ffd55d6b-1688-46f3-b2e1-7c68f86b908d\") " pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.554461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.554059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffd55d6b-1688-46f3-b2e1-7c68f86b908d-bound-sa-token\") pod \"cert-manager-55b4d954d5-chz7z\" (UID: \"ffd55d6b-1688-46f3-b2e1-7c68f86b908d\") " pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.565127 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.565096 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffd55d6b-1688-46f3-b2e1-7c68f86b908d-bound-sa-token\") pod \"cert-manager-55b4d954d5-chz7z\" (UID: \"ffd55d6b-1688-46f3-b2e1-7c68f86b908d\") " pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.565417 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.565398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66th\" (UniqueName: \"kubernetes.io/projected/ffd55d6b-1688-46f3-b2e1-7c68f86b908d-kube-api-access-d66th\") pod \"cert-manager-55b4d954d5-chz7z\" (UID: \"ffd55d6b-1688-46f3-b2e1-7c68f86b908d\") " pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.578175 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.578150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-chz7z" Jul 10 16:00:03.703809 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:03.703717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-chz7z"] Jul 10 16:00:03.707269 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:00:03.707220 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffd55d6b_1688_46f3_b2e1_7c68f86b908d.slice/crio-136cd3ad63be89036e6e2dec77905d6c3c5a75c10b0fbcaf20cbf6fbc9f87b71 WatchSource:0}: Error finding container 136cd3ad63be89036e6e2dec77905d6c3c5a75c10b0fbcaf20cbf6fbc9f87b71: Status 404 returned error can't find the container with id 136cd3ad63be89036e6e2dec77905d6c3c5a75c10b0fbcaf20cbf6fbc9f87b71 Jul 10 16:00:04.690133 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:04.690103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-chz7z" event={"ID":"ffd55d6b-1688-46f3-b2e1-7c68f86b908d","Type":"ContainerStarted","Data":"136cd3ad63be89036e6e2dec77905d6c3c5a75c10b0fbcaf20cbf6fbc9f87b71"} Jul 10 16:00:05.863410 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:05.863377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:08.707887 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:08.707847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-chz7z" event={"ID":"ffd55d6b-1688-46f3-b2e1-7c68f86b908d","Type":"ContainerStarted","Data":"d9a41493dc5bfd047f160b90b50dee42d81e40c54fd62050a315696dc686f2d7"} Jul 10 16:00:08.726975 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:08.726907 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-chz7z" podStartSLOduration=1.14575206 podStartE2EDuration="5.726889502s" totalImagesPullingTime="4.581137442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:00:08.725572722 +0000 UTC m=+297.796390820" watchObservedRunningTime="2026-07-10 16:00:08.726889502 +0000 UTC m=+297.797707573" Jul 10 16:00:10.197211 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:10.197178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:10.298113 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:10.298062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:11.323648 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:11.323605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:00:11.326640 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:11.326618 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:00:11.327628 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:11.327612 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:00:11.331032 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:11.331015 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:00:11.335107 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:11.335090 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 16:00:11.578208 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:11.577996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:12.337899 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:12.337859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:23.911885 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:23.911848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.050781 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.050738 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j"] Jul 10 16:00:25.052822 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.052804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.055446 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.055420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 16:00:25.055731 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.055714 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 16:00:25.055794 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.055739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 16:00:25.055848 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.055795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5fx84\"" Jul 10 16:00:25.056015 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.056002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 16:00:25.075194 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.075161 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j"] Jul 10 16:00:25.116878 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.116840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5676-7790-41fb-883d-55bf43d1d7f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.116878 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.116878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea0f5676-7790-41fb-883d-55bf43d1d7f1-manifests\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.117101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.116916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5676-7790-41fb-883d-55bf43d1d7f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.117101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.116947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swk5l\" (UniqueName: \"kubernetes.io/projected/ea0f5676-7790-41fb-883d-55bf43d1d7f1-kube-api-access-swk5l\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.117101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.116983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea0f5676-7790-41fb-883d-55bf43d1d7f1-tmp\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.217744 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.217709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea0f5676-7790-41fb-883d-55bf43d1d7f1-tmp\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.217919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.217762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5676-7790-41fb-883d-55bf43d1d7f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.217919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.217786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea0f5676-7790-41fb-883d-55bf43d1d7f1-manifests\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.217919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.217817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5676-7790-41fb-883d-55bf43d1d7f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.217919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.217837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swk5l\" (UniqueName: \"kubernetes.io/projected/ea0f5676-7790-41fb-883d-55bf43d1d7f1-kube-api-access-swk5l\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.218362 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.218336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea0f5676-7790-41fb-883d-55bf43d1d7f1-manifests\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.220467 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.220446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5676-7790-41fb-883d-55bf43d1d7f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.220673 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.220655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea0f5676-7790-41fb-883d-55bf43d1d7f1-tmp\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.220793 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.220775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5676-7790-41fb-883d-55bf43d1d7f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.226958 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.226932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swk5l\" (UniqueName: \"kubernetes.io/projected/ea0f5676-7790-41fb-883d-55bf43d1d7f1-kube-api-access-swk5l\") pod \"opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j\" (UID: \"ea0f5676-7790-41fb-883d-55bf43d1d7f1\") " pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.316653 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.316553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.362438 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.362406 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:25.413492 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.413459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.499554 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.499527 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j"] Jul 10 16:00:25.502089 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:00:25.502062 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0f5676_7790_41fb_883d_55bf43d1d7f1.slice/crio-f11224e8017eefc7f006411e51fee8e21456cfc4f2c0a842910e63806c88fe6d WatchSource:0}: Error finding container f11224e8017eefc7f006411e51fee8e21456cfc4f2c0a842910e63806c88fe6d: Status 404 returned error can't find the container with id f11224e8017eefc7f006411e51fee8e21456cfc4f2c0a842910e63806c88fe6d Jul 10 16:00:25.504123 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.504105 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:00:25.507500 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.507472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.580334 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.580231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.681593 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.681551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.771237 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.771204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" event={"ID":"ea0f5676-7790-41fb-883d-55bf43d1d7f1","Type":"ContainerStarted","Data":"f11224e8017eefc7f006411e51fee8e21456cfc4f2c0a842910e63806c88fe6d"} Jul 10 16:00:25.779991 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.779961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:25.880205 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:25.880108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:26.664205 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:26.664144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:26.763131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:26.763092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:28.785593 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:28.785503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" event={"ID":"ea0f5676-7790-41fb-883d-55bf43d1d7f1","Type":"ContainerStarted","Data":"f489b2d06933ce3585dea33fe3882df4636ae12ff69d57564a8655a6929ff392"} Jul 10 16:00:29.790799 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:29.790763 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea0f5676-7790-41fb-883d-55bf43d1d7f1" containerID="f489b2d06933ce3585dea33fe3882df4636ae12ff69d57564a8655a6929ff392" exitCode=0 Jul 10 16:00:29.791184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:29.790813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" event={"ID":"ea0f5676-7790-41fb-883d-55bf43d1d7f1","Type":"ContainerDied","Data":"f489b2d06933ce3585dea33fe3882df4636ae12ff69d57564a8655a6929ff392"} Jul 10 16:00:30.798194 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:30.798159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" event={"ID":"ea0f5676-7790-41fb-883d-55bf43d1d7f1","Type":"ContainerStarted","Data":"8958df90653d3e0957365d153288a82bf6d25f7fb20ccd4dabb717ebbdf7ff2d"} Jul 10 16:00:30.798690 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:30.798266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:30.822019 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:30.821964 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" podStartSLOduration=1.446982205 podStartE2EDuration="5.821948821s" totalImagesPullingTime="3.296582963s" totalInitContainerRuntime="1.078383653s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:00:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:00:30.820706432 +0000 UTC m=+319.891524526" watchObservedRunningTime="2026-07-10 16:00:30.821948821 +0000 UTC m=+319.892766891" Jul 10 16:00:41.804361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:41.804327 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j" Jul 10 16:00:44.475455 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:44.475402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:44.571918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:44.571879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:45.009166 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.009124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:45.177371 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.177335 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64"] Jul 10 16:00:45.188260 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.188216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.190673 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.190646 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 16:00:45.190806 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.190672 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 16:00:45.190950 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.190934 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 16:00:45.191284 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.191268 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 16:00:45.191375 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.191309 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xltxs\"" Jul 10 16:00:45.195386 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.195286 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64"] Jul 10 16:00:45.280967 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.280874 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v8ffm"] Jul 10 16:00:45.282919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.282894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zl7\" (UniqueName: \"kubernetes.io/projected/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-kube-api-access-t6zl7\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.283027 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.282975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-tls-certs\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.283180 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.283143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-tmp\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.283414 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.283395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.285697 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.285669 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sh76h\"" Jul 10 16:00:45.285697 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.285667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 16:00:45.288475 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.288447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v8ffm"] Jul 10 16:00:45.384383 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.384343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-tls-certs\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.384383 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.384397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dbcfb7c-695b-471f-b775-79fbeccb1d59-cert\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.384650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.384435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-tmp\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.384650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.384524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zl7\" (UniqueName: \"kubernetes.io/projected/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-kube-api-access-t6zl7\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.384650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.384582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4dl\" (UniqueName: \"kubernetes.io/projected/0dbcfb7c-695b-471f-b775-79fbeccb1d59-kube-api-access-9f4dl\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.387239 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.387212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-tmp\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.387427 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.387405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-tls-certs\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.393483 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.393452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zl7\" (UniqueName: \"kubernetes.io/projected/2714aff7-5112-4a2a-8a79-7fc66d6c52d3-kube-api-access-t6zl7\") pod \"kube-auth-proxy-8457fcdff5-k6w64\" (UID: \"2714aff7-5112-4a2a-8a79-7fc66d6c52d3\") " pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.485766 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.485731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4dl\" (UniqueName: \"kubernetes.io/projected/0dbcfb7c-695b-471f-b775-79fbeccb1d59-kube-api-access-9f4dl\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.486199 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.485799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dbcfb7c-695b-471f-b775-79fbeccb1d59-cert\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.486199 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:00:45.485913 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 16:00:45.486199 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:00:45.485978 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbcfb7c-695b-471f-b775-79fbeccb1d59-cert podName:0dbcfb7c-695b-471f-b775-79fbeccb1d59 nodeName:}" failed. No retries permitted until 2026-07-10 16:00:45.985961696 +0000 UTC m=+335.056779745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0dbcfb7c-695b-471f-b775-79fbeccb1d59-cert") pod "odh-model-controller-7998c67dc4-v8ffm" (UID: "0dbcfb7c-695b-471f-b775-79fbeccb1d59") : secret "odh-model-controller-webhook-cert" not found Jul 10 16:00:45.494487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.494450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4dl\" (UniqueName: \"kubernetes.io/projected/0dbcfb7c-695b-471f-b775-79fbeccb1d59-kube-api-access-9f4dl\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.499654 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.499627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" Jul 10 16:00:45.658172 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.658149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64"] Jul 10 16:00:45.660749 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:00:45.660721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2714aff7_5112_4a2a_8a79_7fc66d6c52d3.slice/crio-17e175ed2d8cc9bbf895b895e213b4e03749d24930e188106e958016272ede77 WatchSource:0}: Error finding container 17e175ed2d8cc9bbf895b895e213b4e03749d24930e188106e958016272ede77: Status 404 returned error can't find the container with id 17e175ed2d8cc9bbf895b895e213b4e03749d24930e188106e958016272ede77 Jul 10 16:00:45.859762 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.859671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" event={"ID":"2714aff7-5112-4a2a-8a79-7fc66d6c52d3","Type":"ContainerStarted","Data":"17e175ed2d8cc9bbf895b895e213b4e03749d24930e188106e958016272ede77"} Jul 10 16:00:45.990619 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.990570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dbcfb7c-695b-471f-b775-79fbeccb1d59-cert\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:45.993370 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:45.993343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dbcfb7c-695b-471f-b775-79fbeccb1d59-cert\") pod \"odh-model-controller-7998c67dc4-v8ffm\" (UID: \"0dbcfb7c-695b-471f-b775-79fbeccb1d59\") " pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:46.196756 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.196721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:46.328425 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.328389 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx"] Jul 10 16:00:46.331167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.331141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.333341 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.333319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 16:00:46.333571 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.333319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pc6xp\"" Jul 10 16:00:46.333959 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.333767 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 16:00:46.333959 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.333781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 16:00:46.333959 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.333863 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 16:00:46.352843 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.352818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v8ffm"] Jul 10 16:00:46.354401 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.354372 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx"] Jul 10 16:00:46.355453 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:00:46.355427 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbcfb7c_695b_471f_b775_79fbeccb1d59.slice/crio-acb38fd4879240b39fffc30243b3c951472b6a0b5dfd9297b5048dd56d8eff19 WatchSource:0}: Error finding container acb38fd4879240b39fffc30243b3c951472b6a0b5dfd9297b5048dd56d8eff19: Status 404 returned error can't find the container with id acb38fd4879240b39fffc30243b3c951472b6a0b5dfd9297b5048dd56d8eff19 Jul 10 16:00:46.393603 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-token\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.393768 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-cacerts\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.393768 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8abcc0-f59a-4299-a574-3db5d159ce29-local-certs\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.393768 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.394010 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26g9f\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-kube-api-access-26g9f\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.394010 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.394010 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.393928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.418310 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.418271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:46.495319 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.495319 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26g9f\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-kube-api-access-26g9f\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.495829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.495829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.495829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-token\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.495829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-cacerts\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.495829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8abcc0-f59a-4299-a574-3db5d159ce29-local-certs\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.496076 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.495863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.499085 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.498934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-cacerts\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.499085 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.499038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.499300 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.499126 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.499643 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.499596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8abcc0-f59a-4299-a574-3db5d159ce29-local-certs\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.504887 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.504858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-token\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.505738 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.505691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26g9f\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-kube-api-access-26g9f\") pod \"istiod-openshift-gateway-7574459656-jx2hx\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.519506 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.519470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:46.642615 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.642585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:46.722941 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.722892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:46.796772 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.796685 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx"] Jul 10 16:00:46.800239 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:00:46.800206 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8abcc0_f59a_4299_a574_3db5d159ce29.slice/crio-11e6b75d7c6f5c5ca2a4a5983094e7055feb912457b1e333b3dad44a23836410 WatchSource:0}: Error finding container 11e6b75d7c6f5c5ca2a4a5983094e7055feb912457b1e333b3dad44a23836410: Status 404 returned error can't find the container with id 11e6b75d7c6f5c5ca2a4a5983094e7055feb912457b1e333b3dad44a23836410 Jul 10 16:00:46.865667 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.865632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" event={"ID":"8d8abcc0-f59a-4299-a574-3db5d159ce29","Type":"ContainerStarted","Data":"11e6b75d7c6f5c5ca2a4a5983094e7055feb912457b1e333b3dad44a23836410"} Jul 10 16:00:46.866774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:46.866751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" event={"ID":"0dbcfb7c-695b-471f-b775-79fbeccb1d59","Type":"ContainerStarted","Data":"acb38fd4879240b39fffc30243b3c951472b6a0b5dfd9297b5048dd56d8eff19"} Jul 10 16:00:47.448930 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:47.448885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:51.205021 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.204981 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:00:51.205446 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.205062 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:00:51.896210 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.895983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" event={"ID":"8d8abcc0-f59a-4299-a574-3db5d159ce29","Type":"ContainerStarted","Data":"d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567"} Jul 10 16:00:51.896210 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.896091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:51.897873 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.897839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" event={"ID":"0dbcfb7c-695b-471f-b775-79fbeccb1d59","Type":"ContainerStarted","Data":"c067aac35ee9cb65d39fb2a46fa27eb1b9836d230393bbd00de547a19277abb4"} Jul 10 16:00:51.897997 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.897906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:00:51.899801 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.899777 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" event={"ID":"2714aff7-5112-4a2a-8a79-7fc66d6c52d3","Type":"ContainerStarted","Data":"b42671550eea4567df2c11c1154a6bcab2e3b7bcff3dd9613b2b44cc3ab261d3"} Jul 10 16:00:51.921032 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.920970 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" podStartSLOduration=1.519027007 podStartE2EDuration="5.920953618s" totalImagesPullingTime="4.401926611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:00:51.919448716 +0000 UTC m=+340.990266787" watchObservedRunningTime="2026-07-10 16:00:51.920953618 +0000 UTC m=+340.991771679" Jul 10 16:00:51.937999 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.937936 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8457fcdff5-k6w64" podStartSLOduration=1.391005564 podStartE2EDuration="6.937916875s" totalImagesPullingTime="5.546911311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:00:51.937367636 +0000 UTC m=+341.008185719" watchObservedRunningTime="2026-07-10 16:00:51.937916875 +0000 UTC m=+341.008734947" Jul 10 16:00:51.966825 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:51.966768 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" podStartSLOduration=2.121811735 podStartE2EDuration="6.966745875s" totalImagesPullingTime="4.84493414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:00:51.955150159 +0000 UTC m=+341.025968256" watchObservedRunningTime="2026-07-10 16:00:51.966745875 +0000 UTC m=+341.037563946" Jul 10 16:00:52.031139 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:52.031092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:52.095073 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:52.095031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:52.889991 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:52.889948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:52.905791 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:52.905767 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:00:53.017120 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:53.017075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:53.233138 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:53.233100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:00:57.856211 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:57.856175 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78"] Jul 10 16:00:57.859131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:57.859115 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:57.919945 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:57.919909 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78"] Jul 10 16:00:58.003687 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.003687 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.003926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.003926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc98l\" (UniqueName: \"kubernetes.io/projected/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-kube-api-access-cc98l\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.003926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.003926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.003926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.003890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105037 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.104997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.105051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.105070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.105089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.105108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc98l\" (UniqueName: \"kubernetes.io/projected/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-kube-api-access-cc98l\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.105131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.105193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.105162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.106188 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.106156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.107488 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.107429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.107741 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.107724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.107786 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.107725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.107820 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.107795 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.126047 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.126023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.127669 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.127645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc98l\" (UniqueName: \"kubernetes.io/projected/c5c36812-f887-4b3a-acd0-c77dbd9c12dc-kube-api-access-cc98l\") pod \"istiod-openshift-gateway-94bb8fbfd-hpm78\" (UID: \"c5c36812-f887-4b3a-acd0-c77dbd9c12dc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.167664 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.167628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.345552 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.345388 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78"] Jul 10 16:00:58.348204 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:00:58.348178 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c36812_f887_4b3a_acd0_c77dbd9c12dc.slice/crio-3b93bd4925e38dcdc8b4807543fd7e27b2b3deda8b9b760b26646f1879d561af WatchSource:0}: Error finding container 3b93bd4925e38dcdc8b4807543fd7e27b2b3deda8b9b760b26646f1879d561af: Status 404 returned error can't find the container with id 3b93bd4925e38dcdc8b4807543fd7e27b2b3deda8b9b760b26646f1879d561af Jul 10 16:00:58.351029 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.350998 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:00:58.351117 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.351062 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:00:58.931952 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.931914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" event={"ID":"c5c36812-f887-4b3a-acd0-c77dbd9c12dc","Type":"ContainerStarted","Data":"4b1f2bf0eae786269ed5b5c127eae2a00861b66a0a106a4b0a9a1370acc3b45f"} Jul 10 16:00:58.932476 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.931973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" event={"ID":"c5c36812-f887-4b3a-acd0-c77dbd9c12dc","Type":"ContainerStarted","Data":"3b93bd4925e38dcdc8b4807543fd7e27b2b3deda8b9b760b26646f1879d561af"} Jul 10 16:00:58.932476 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.932014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:00:58.982606 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:58.982547 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" podStartSLOduration=1.982525477 podStartE2EDuration="1.982525477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:00:58.981515022 +0000 UTC m=+348.052333094" watchObservedRunningTime="2026-07-10 16:00:58.982525477 +0000 UTC m=+348.053343546" Jul 10 16:00:59.938670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:00:59.938637 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hpm78" Jul 10 16:01:00.105151 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.105091 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx"] Jul 10 16:01:00.105396 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.105341 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" podUID="8d8abcc0-f59a-4299-a574-3db5d159ce29" containerName="discovery" containerID="cri-o://d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567" gracePeriod=30 Jul 10 16:01:00.352981 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.352958 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:01:00.427487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427457 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-ca-configmap\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427508 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-token\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427537 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-cacerts\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427577 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8abcc0-f59a-4299-a574-3db5d159ce29-local-certs\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-kubeconfig\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427653 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-dns-cert\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427955 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427707 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26g9f\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-kube-api-access-26g9f\") pod \"8d8abcc0-f59a-4299-a574-3db5d159ce29\" (UID: \"8d8abcc0-f59a-4299-a574-3db5d159ce29\") " Jul 10 16:01:00.427955 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.427924 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:01:00.430934 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.430531 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-cacerts" (OuterVolumeSpecName: "cacerts") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:01:00.431069 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.431000 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:01:00.431217 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.431178 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:01:00.431436 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.431328 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8abcc0-f59a-4299-a574-3db5d159ce29-local-certs" (OuterVolumeSpecName: "local-certs") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:01:00.431436 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.431337 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:01:00.431702 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.431640 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-kube-api-access-26g9f" (OuterVolumeSpecName: "kube-api-access-26g9f") pod "8d8abcc0-f59a-4299-a574-3db5d159ce29" (UID: "8d8abcc0-f59a-4299-a574-3db5d159ce29"). InnerVolumeSpecName "kube-api-access-26g9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.529871 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26g9f\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-kube-api-access-26g9f\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.529926 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-ca-configmap\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.529958 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-token\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.529995 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-cacerts\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.530017 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8abcc0-f59a-4299-a574-3db5d159ce29-local-certs\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.530035 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-kubeconfig\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.534473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.530053 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d8abcc0-f59a-4299-a574-3db5d159ce29-istio-csr-dns-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:00.942339 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.942302 2576 generic.go:356] "Generic (PLEG): container finished" podID="8d8abcc0-f59a-4299-a574-3db5d159ce29" containerID="d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567" exitCode=0 Jul 10 16:01:00.942774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.942367 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" Jul 10 16:01:00.942774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.942390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" event={"ID":"8d8abcc0-f59a-4299-a574-3db5d159ce29","Type":"ContainerDied","Data":"d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567"} Jul 10 16:01:00.942774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.942506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx" event={"ID":"8d8abcc0-f59a-4299-a574-3db5d159ce29","Type":"ContainerDied","Data":"11e6b75d7c6f5c5ca2a4a5983094e7055feb912457b1e333b3dad44a23836410"} Jul 10 16:01:00.942774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.942533 2576 scope.go:117] "RemoveContainer" containerID="d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567" Jul 10 16:01:00.951367 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.951350 2576 scope.go:117] "RemoveContainer" containerID="d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567" Jul 10 16:01:00.951664 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:01:00.951644 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567\": container with ID starting with d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567 not found: ID does not exist" containerID="d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567" Jul 10 16:01:00.951709 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.951675 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567"} err="failed to get container status \"d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567\": rpc error: code = NotFound desc = could not find container \"d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567\": container with ID starting with d91375499d7d2f35f79153261b2e757fbd017687a2f015e52104eb483e5dc567 not found: ID does not exist" Jul 10 16:01:00.995706 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:00.995674 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx"] Jul 10 16:01:01.024386 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.024356 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx2hx"] Jul 10 16:01:01.334406 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.334320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:01.485133 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.485092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:01.495043 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.495017 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d8abcc0-f59a-4299-a574-3db5d159ce29" path="/var/lib/kubelet/pods/8d8abcc0-f59a-4299-a574-3db5d159ce29/volumes" Jul 10 16:01:01.528508 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.528472 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7"] Jul 10 16:01:01.528948 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.528935 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d8abcc0-f59a-4299-a574-3db5d159ce29" containerName="discovery" Jul 10 16:01:01.528991 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.528952 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d8abcc0-f59a-4299-a574-3db5d159ce29" containerName="discovery" Jul 10 16:01:01.529055 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.529024 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d8abcc0-f59a-4299-a574-3db5d159ce29" containerName="discovery" Jul 10 16:01:01.531907 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.531884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.549748 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.549721 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 16:01:01.554886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.554862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7"] Jul 10 16:01:01.556608 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.556585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-5vdk9\"" Jul 10 16:01:01.635336 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.635200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:01.639301 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.639271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnrdz\" (UniqueName: \"kubernetes.io/projected/bc08ecc2-5b93-4b46-8319-923bc7c4f4e7-kube-api-access-lnrdz\") pod \"kserve-controller-manager-d6f4b65b8-4nlh7\" (UID: \"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.639449 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.639306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc08ecc2-5b93-4b46-8319-923bc7c4f4e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-4nlh7\" (UID: \"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.739882 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.739846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnrdz\" (UniqueName: \"kubernetes.io/projected/bc08ecc2-5b93-4b46-8319-923bc7c4f4e7-kube-api-access-lnrdz\") pod \"kserve-controller-manager-d6f4b65b8-4nlh7\" (UID: \"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.739882 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.739883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc08ecc2-5b93-4b46-8319-923bc7c4f4e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-4nlh7\" (UID: \"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.742605 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.742579 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc08ecc2-5b93-4b46-8319-923bc7c4f4e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-4nlh7\" (UID: \"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.752874 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.752847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:01.754419 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.754397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnrdz\" (UniqueName: \"kubernetes.io/projected/bc08ecc2-5b93-4b46-8319-923bc7c4f4e7-kube-api-access-lnrdz\") pod \"kserve-controller-manager-d6f4b65b8-4nlh7\" (UID: \"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.842607 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.842571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:01.853360 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.853316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:01.998456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:01.998422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:02.000868 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:01:02.000843 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc08ecc2_5b93_4b46_8319_923bc7c4f4e7.slice/crio-e88a2a7b487c9c4345ee6c5dde9b71280d5d616aa8d0541263f3914db916ef19 WatchSource:0}: Error finding container e88a2a7b487c9c4345ee6c5dde9b71280d5d616aa8d0541263f3914db916ef19: Status 404 returned error can't find the container with id e88a2a7b487c9c4345ee6c5dde9b71280d5d616aa8d0541263f3914db916ef19 Jul 10 16:01:02.001221 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:02.001200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7"] Jul 10 16:01:02.130691 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:02.130652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:02.907502 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:02.907466 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v8ffm" Jul 10 16:01:02.954503 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:02.954464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" event={"ID":"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7","Type":"ContainerStarted","Data":"e88a2a7b487c9c4345ee6c5dde9b71280d5d616aa8d0541263f3914db916ef19"} Jul 10 16:01:04.965031 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:04.964986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" event={"ID":"bc08ecc2-5b93-4b46-8319-923bc7c4f4e7","Type":"ContainerStarted","Data":"ec02228dbf71e2285ea844bffb1c63eb2c15c3b02c740f0ec86c5ca0e3a08065"} Jul 10 16:01:04.965415 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:04.965102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:05.001241 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:05.001133 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" podStartSLOduration=1.26502688 podStartE2EDuration="4.001117876s" totalImagesPullingTime="2.736090996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:01:05.000478544 +0000 UTC m=+354.071296641" watchObservedRunningTime="2026-07-10 16:01:05.001117876 +0000 UTC m=+354.071935947" Jul 10 16:01:31.656168 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:31.656128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:31.717645 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:31.717607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:35.974395 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:35.974365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4nlh7" Jul 10 16:01:45.996713 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:45.996668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:46.102203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.102154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:46.157894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.157854 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l79wc"] Jul 10 16:01:46.160001 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.159985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:46.162405 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.162381 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 16:01:46.162534 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.162381 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-mnqff\"" Jul 10 16:01:46.162596 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.162567 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 16:01:46.171752 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.171716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l79wc"] Jul 10 16:01:46.207473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.207441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:01:46.230553 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.230509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtb5\" (UniqueName: \"kubernetes.io/projected/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc-kube-api-access-frtb5\") pod \"kuadrant-operator-catalog-l79wc\" (UID: \"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc\") " pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:46.331226 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.331132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtb5\" (UniqueName: \"kubernetes.io/projected/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc-kube-api-access-frtb5\") pod \"kuadrant-operator-catalog-l79wc\" (UID: \"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc\") " pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:46.339331 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.339296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtb5\" (UniqueName: \"kubernetes.io/projected/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc-kube-api-access-frtb5\") pod \"kuadrant-operator-catalog-l79wc\" (UID: \"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc\") " pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:46.472054 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.472015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:46.521338 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.520878 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l79wc"] Jul 10 16:01:46.609686 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.609605 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l79wc"] Jul 10 16:01:46.613670 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:01:46.613643 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd176bd43_b6bd_4ec9_9ea9_cbcc0ff0f5bc.slice/crio-9f3b8915d23c7c70f2bb4e5dc3a3f44027d9a03c890e3f71f23c84d63221a195 WatchSource:0}: Error finding container 9f3b8915d23c7c70f2bb4e5dc3a3f44027d9a03c890e3f71f23c84d63221a195: Status 404 returned error can't find the container with id 9f3b8915d23c7c70f2bb4e5dc3a3f44027d9a03c890e3f71f23c84d63221a195 Jul 10 16:01:46.730128 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.730089 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7pnr"] Jul 10 16:01:46.732954 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.732931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:46.742421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.742381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7pnr"] Jul 10 16:01:46.835672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.835631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgwq\" (UniqueName: \"kubernetes.io/projected/bef8df15-9934-424c-967f-8bef761df49e-kube-api-access-dtgwq\") pod \"kuadrant-operator-catalog-p7pnr\" (UID: \"bef8df15-9934-424c-967f-8bef761df49e\") " pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:46.936315 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.936279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgwq\" (UniqueName: \"kubernetes.io/projected/bef8df15-9934-424c-967f-8bef761df49e-kube-api-access-dtgwq\") pod \"kuadrant-operator-catalog-p7pnr\" (UID: \"bef8df15-9934-424c-967f-8bef761df49e\") " pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:46.945792 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:46.945763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgwq\" (UniqueName: \"kubernetes.io/projected/bef8df15-9934-424c-967f-8bef761df49e-kube-api-access-dtgwq\") pod \"kuadrant-operator-catalog-p7pnr\" (UID: \"bef8df15-9934-424c-967f-8bef761df49e\") " pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:47.042857 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:47.042819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:47.151463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:47.151415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" event={"ID":"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc","Type":"ContainerStarted","Data":"9f3b8915d23c7c70f2bb4e5dc3a3f44027d9a03c890e3f71f23c84d63221a195"} Jul 10 16:01:47.178628 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:47.178604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7pnr"] Jul 10 16:01:47.180489 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:01:47.180467 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef8df15_9934_424c_967f_8bef761df49e.slice/crio-41c108039f5b4906bd7b790bea055cd5ddc098ee0552bfad0fb5825187b54439 WatchSource:0}: Error finding container 41c108039f5b4906bd7b790bea055cd5ddc098ee0552bfad0fb5825187b54439: Status 404 returned error can't find the container with id 41c108039f5b4906bd7b790bea055cd5ddc098ee0552bfad0fb5825187b54439 Jul 10 16:01:48.158956 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:48.158921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" event={"ID":"bef8df15-9934-424c-967f-8bef761df49e","Type":"ContainerStarted","Data":"41c108039f5b4906bd7b790bea055cd5ddc098ee0552bfad0fb5825187b54439"} Jul 10 16:01:49.165062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.164967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" event={"ID":"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc","Type":"ContainerStarted","Data":"b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2"} Jul 10 16:01:49.165062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.165044 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" podUID="d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" containerName="registry-server" containerID="cri-o://b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2" gracePeriod=2 Jul 10 16:01:49.182731 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.182685 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" podStartSLOduration=0.93760818 podStartE2EDuration="3.182666455s" totalImagesPullingTime="2.245058275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:01:49.181698312 +0000 UTC m=+398.252516382" watchObservedRunningTime="2026-07-10 16:01:49.182666455 +0000 UTC m=+398.253484525" Jul 10 16:01:49.409472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.409448 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:49.563193 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.563161 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frtb5\" (UniqueName: \"kubernetes.io/projected/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc-kube-api-access-frtb5\") pod \"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc\" (UID: \"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc\") " Jul 10 16:01:49.565487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.565459 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc-kube-api-access-frtb5" (OuterVolumeSpecName: "kube-api-access-frtb5") pod "d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" (UID: "d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc"). InnerVolumeSpecName "kube-api-access-frtb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:01:49.663862 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:49.663824 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtb5\" (UniqueName: \"kubernetes.io/projected/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc-kube-api-access-frtb5\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:01:50.170896 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.170793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" event={"ID":"bef8df15-9934-424c-967f-8bef761df49e","Type":"ContainerStarted","Data":"a5987565e08c5b9ff00accf5c11882639ee7e30101896b0eccd50e8bc52e17b8"} Jul 10 16:01:50.171970 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.171953 2576 generic.go:356] "Generic (PLEG): container finished" podID="d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" containerID="b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2" exitCode=0 Jul 10 16:01:50.172035 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.171982 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" Jul 10 16:01:50.172035 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.172016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" event={"ID":"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc","Type":"ContainerDied","Data":"b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2"} Jul 10 16:01:50.172116 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.172048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l79wc" event={"ID":"d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc","Type":"ContainerDied","Data":"9f3b8915d23c7c70f2bb4e5dc3a3f44027d9a03c890e3f71f23c84d63221a195"} Jul 10 16:01:50.172116 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.172067 2576 scope.go:117] "RemoveContainer" containerID="b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2" Jul 10 16:01:50.181024 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.181007 2576 scope.go:117] "RemoveContainer" containerID="b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2" Jul 10 16:01:50.181318 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:01:50.181299 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2\": container with ID starting with b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2 not found: ID does not exist" containerID="b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2" Jul 10 16:01:50.181393 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.181325 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2"} err="failed to get container status \"b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2\": rpc error: code = NotFound desc = could not find container \"b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2\": container with ID starting with b9e7447300b1f3e2ad2ffe5dce15b6d6268d4dc12a0c48a05771eb66de73cff2 not found: ID does not exist" Jul 10 16:01:50.187715 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.187672 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" podStartSLOduration=1.72580701 podStartE2EDuration="4.187658911s" totalImagesPullingTime="2.461851901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:01:50.186116126 +0000 UTC m=+399.256934214" watchObservedRunningTime="2026-07-10 16:01:50.187658911 +0000 UTC m=+399.258476980" Jul 10 16:01:50.198020 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.197994 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l79wc"] Jul 10 16:01:50.201595 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:50.201568 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l79wc"] Jul 10 16:01:51.494989 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:51.494957 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" path="/var/lib/kubelet/pods/d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc/volumes" Jul 10 16:01:57.043341 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:57.043301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:57.043766 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:57.043393 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:57.066222 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:57.066193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:01:57.224361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:01:57.224330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-p7pnr" Jul 10 16:02:14.962071 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:14.962021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:17.626647 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.626608 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc"] Jul 10 16:02:17.627020 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.626936 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" containerName="registry-server" Jul 10 16:02:17.627020 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.626947 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" containerName="registry-server" Jul 10 16:02:17.627020 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.627002 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d176bd43-b6bd-4ec9-9ea9-cbcc0ff0f5bc" containerName="registry-server" Jul 10 16:02:17.631081 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.631064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:17.633472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.633449 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 16:02:17.633610 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.633496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-llbcw\"" Jul 10 16:02:17.640921 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.640899 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc"] Jul 10 16:02:17.688340 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.688308 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2m26\" (UniqueName: \"kubernetes.io/projected/f81f11a3-e278-4b28-a195-3c89327c8466-kube-api-access-v2m26\") pod \"dns-operator-controller-manager-864dd78c4c-rrpnc\" (UID: \"f81f11a3-e278-4b28-a195-3c89327c8466\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:17.788879 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.788837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2m26\" (UniqueName: \"kubernetes.io/projected/f81f11a3-e278-4b28-a195-3c89327c8466-kube-api-access-v2m26\") pod \"dns-operator-controller-manager-864dd78c4c-rrpnc\" (UID: \"f81f11a3-e278-4b28-a195-3c89327c8466\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:17.797330 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.797300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2m26\" (UniqueName: \"kubernetes.io/projected/f81f11a3-e278-4b28-a195-3c89327c8466-kube-api-access-v2m26\") pod \"dns-operator-controller-manager-864dd78c4c-rrpnc\" (UID: \"f81f11a3-e278-4b28-a195-3c89327c8466\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:17.941654 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:17.941624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:18.072240 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:18.072215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc"] Jul 10 16:02:18.074536 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:02:18.074509 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81f11a3_e278_4b28_a195_3c89327c8466.slice/crio-4472509fd188138391f65b1f916f0bf53767cf9ffff1fad2f00d7da711e11cf1 WatchSource:0}: Error finding container 4472509fd188138391f65b1f916f0bf53767cf9ffff1fad2f00d7da711e11cf1: Status 404 returned error can't find the container with id 4472509fd188138391f65b1f916f0bf53767cf9ffff1fad2f00d7da711e11cf1 Jul 10 16:02:18.305361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:18.305278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" event={"ID":"f81f11a3-e278-4b28-a195-3c89327c8466","Type":"ContainerStarted","Data":"4472509fd188138391f65b1f916f0bf53767cf9ffff1fad2f00d7da711e11cf1"} Jul 10 16:02:19.186365 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:19.186331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:21.035493 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.035462 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89"] Jul 10 16:02:21.038759 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.038744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:21.040826 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.040802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zxngh\"" Jul 10 16:02:21.047658 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.047634 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89"] Jul 10 16:02:21.118334 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.118287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcsn\" (UniqueName: \"kubernetes.io/projected/0141bc89-387e-4a08-ac8f-6fcb997bf471-kube-api-access-5rcsn\") pod \"limitador-operator-controller-manager-859d7b78bd-86z89\" (UID: \"0141bc89-387e-4a08-ac8f-6fcb997bf471\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:21.219814 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.219785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcsn\" (UniqueName: \"kubernetes.io/projected/0141bc89-387e-4a08-ac8f-6fcb997bf471-kube-api-access-5rcsn\") pod \"limitador-operator-controller-manager-859d7b78bd-86z89\" (UID: \"0141bc89-387e-4a08-ac8f-6fcb997bf471\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:21.230523 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.230497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcsn\" (UniqueName: \"kubernetes.io/projected/0141bc89-387e-4a08-ac8f-6fcb997bf471-kube-api-access-5rcsn\") pod \"limitador-operator-controller-manager-859d7b78bd-86z89\" (UID: \"0141bc89-387e-4a08-ac8f-6fcb997bf471\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:21.352237 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.352141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:21.541203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.541158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:21.766604 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.766575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:21.780794 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:21.780769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89"] Jul 10 16:02:21.783290 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:02:21.783264 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0141bc89_387e_4a08_ac8f_6fcb997bf471.slice/crio-e3ea1e35329aabfa5f697ddfafa1a101a1090ca1715e363d6f61bb545148f1b1 WatchSource:0}: Error finding container e3ea1e35329aabfa5f697ddfafa1a101a1090ca1715e363d6f61bb545148f1b1: Status 404 returned error can't find the container with id e3ea1e35329aabfa5f697ddfafa1a101a1090ca1715e363d6f61bb545148f1b1 Jul 10 16:02:22.142789 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:22.142753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:22.325536 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:22.325500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" event={"ID":"f81f11a3-e278-4b28-a195-3c89327c8466","Type":"ContainerStarted","Data":"ffbfecee7ffc5b5ab62f04a4616208dc9803d5827227cb1f26616a40e6f3d955"} Jul 10 16:02:22.325738 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:22.325593 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:22.326754 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:22.326732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" event={"ID":"0141bc89-387e-4a08-ac8f-6fcb997bf471","Type":"ContainerStarted","Data":"e3ea1e35329aabfa5f697ddfafa1a101a1090ca1715e363d6f61bb545148f1b1"} Jul 10 16:02:22.362596 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:22.362543 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" podStartSLOduration=1.764074597 podStartE2EDuration="5.362526588s" totalImagesPullingTime="3.598451991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:02:22.360031357 +0000 UTC m=+431.430849418" watchObservedRunningTime="2026-07-10 16:02:22.362526588 +0000 UTC m=+431.433344658" Jul 10 16:02:24.336501 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:24.336468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" event={"ID":"0141bc89-387e-4a08-ac8f-6fcb997bf471","Type":"ContainerStarted","Data":"ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18"} Jul 10 16:02:24.336883 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:24.336651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:24.355302 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:24.355231 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" podStartSLOduration=1.288369559 podStartE2EDuration="3.355215769s" totalImagesPullingTime="2.06684621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:02:24.353829731 +0000 UTC m=+433.424647817" watchObservedRunningTime="2026-07-10 16:02:24.355215769 +0000 UTC m=+433.426033839" Jul 10 16:02:25.534542 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:25.534500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:26.600774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:26.600739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:26.710367 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:26.710233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:28.637158 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:28.637121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:30.045312 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.045279 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x"] Jul 10 16:02:30.048946 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.048927 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.052192 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.052173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h468v\"" Jul 10 16:02:30.063831 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.063795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x"] Jul 10 16:02:30.097317 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.097281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpp9\" (UniqueName: \"kubernetes.io/projected/11341c1e-c842-4df5-93f0-2dc76dc605dd-kube-api-access-rgpp9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.097456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.097333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11341c1e-c842-4df5-93f0-2dc76dc605dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.198436 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.198402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpp9\" (UniqueName: \"kubernetes.io/projected/11341c1e-c842-4df5-93f0-2dc76dc605dd-kube-api-access-rgpp9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.198436 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.198441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11341c1e-c842-4df5-93f0-2dc76dc605dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.198795 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.198779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11341c1e-c842-4df5-93f0-2dc76dc605dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.208235 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.208213 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpp9\" (UniqueName: \"kubernetes.io/projected/11341c1e-c842-4df5-93f0-2dc76dc605dd-kube-api-access-rgpp9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.359757 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.359673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:30.394952 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.394919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:30.501865 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:30.501823 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x"] Jul 10 16:02:30.504701 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:02:30.504676 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11341c1e_c842_4df5_93f0_2dc76dc605dd.slice/crio-6cf5b7add1dee9c20ceca4dead16572c1d59a93b60c6ce941a97566faf1d92ed WatchSource:0}: Error finding container 6cf5b7add1dee9c20ceca4dead16572c1d59a93b60c6ce941a97566faf1d92ed: Status 404 returned error can't find the container with id 6cf5b7add1dee9c20ceca4dead16572c1d59a93b60c6ce941a97566faf1d92ed Jul 10 16:02:31.370879 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:31.370848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" event={"ID":"11341c1e-c842-4df5-93f0-2dc76dc605dd","Type":"ContainerStarted","Data":"6cf5b7add1dee9c20ceca4dead16572c1d59a93b60c6ce941a97566faf1d92ed"} Jul 10 16:02:31.778550 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:31.778510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:31.878316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:31.878280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:33.333819 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:33.333787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rrpnc" Jul 10 16:02:35.344340 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:35.344308 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:35.564698 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:35.564638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:35.664728 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:35.664643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:39.409767 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:39.409729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" event={"ID":"11341c1e-c842-4df5-93f0-2dc76dc605dd","Type":"ContainerStarted","Data":"aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2"} Jul 10 16:02:39.410243 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:39.409876 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:39.431487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:39.431433 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" podStartSLOduration=1.061767565 podStartE2EDuration="9.431417904s" totalImagesPullingTime="8.369650339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:02:39.429535011 +0000 UTC m=+448.500353081" watchObservedRunningTime="2026-07-10 16:02:39.431417904 +0000 UTC m=+448.502235974" Jul 10 16:02:39.743145 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:39.743106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:39.802136 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:39.802096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:40.205427 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:40.205396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:40.296158 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:40.296119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:41.839962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:41.839927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:42.268688 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:42.268626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:42.435906 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:42.435872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:43.948419 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:43.948384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:44.050532 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:44.050492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:50.417192 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:50.417161 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:50.737750 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:50.737712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:50.845529 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:50.845494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:51.611440 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.611402 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9"] Jul 10 16:02:51.615110 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.615092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.626607 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.626579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9"] Jul 10 16:02:51.784752 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.784713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed735498-5602-4266-885c-ff13ab706fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.784933 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.784775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhpm\" (UniqueName: \"kubernetes.io/projected/ed735498-5602-4266-885c-ff13ab706fcc-kube-api-access-rrhpm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.886000 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.885899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed735498-5602-4266-885c-ff13ab706fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.886000 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.885963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhpm\" (UniqueName: \"kubernetes.io/projected/ed735498-5602-4266-885c-ff13ab706fcc-kube-api-access-rrhpm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.886397 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.886374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed735498-5602-4266-885c-ff13ab706fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.894608 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.894588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhpm\" (UniqueName: \"kubernetes.io/projected/ed735498-5602-4266-885c-ff13ab706fcc-kube-api-access-rrhpm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:51.925541 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:51.925509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:52.066974 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.066847 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9"] Jul 10 16:02:52.069922 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:02:52.069892 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded735498_5602_4266_885c_ff13ab706fcc.slice/crio-f0707846b0dd82025e47f8e102b974da46f170d448668a65487498865e35776a WatchSource:0}: Error finding container f0707846b0dd82025e47f8e102b974da46f170d448668a65487498865e35776a: Status 404 returned error can't find the container with id f0707846b0dd82025e47f8e102b974da46f170d448668a65487498865e35776a Jul 10 16:02:52.346646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.346612 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9"] Jul 10 16:02:52.361892 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.361862 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9"] Jul 10 16:02:52.380359 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.380329 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x"] Jul 10 16:02:52.380627 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.380597 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" containerName="manager" containerID="cri-o://aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2" gracePeriod=2 Jul 10 16:02:52.400963 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.400936 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x"] Jul 10 16:02:52.402423 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402405 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc"] Jul 10 16:02:52.402721 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402710 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" containerName="manager" Jul 10 16:02:52.402759 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402723 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" containerName="manager" Jul 10 16:02:52.402759 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402736 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed735498-5602-4266-885c-ff13ab706fcc" containerName="manager" Jul 10 16:02:52.402759 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402741 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed735498-5602-4266-885c-ff13ab706fcc" containerName="manager" Jul 10 16:02:52.402843 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402803 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed735498-5602-4266-885c-ff13ab706fcc" containerName="manager" Jul 10 16:02:52.402843 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.402812 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" containerName="manager" Jul 10 16:02:52.405648 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.405632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.411284 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.411220 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"11341c1e-c842-4df5-93f0-2dc76dc605dd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\": pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.420408 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.420380 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89"] Jul 10 16:02:52.420646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.420617 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" containerName="manager" containerID="cri-o://ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18" gracePeriod=2 Jul 10 16:02:52.427487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.427438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc"] Jul 10 16:02:52.429759 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.429605 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89"] Jul 10 16:02:52.432001 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.431975 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq"] Jul 10 16:02:52.432471 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.432453 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" containerName="manager" Jul 10 16:02:52.432471 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.432471 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" containerName="manager" Jul 10 16:02:52.432619 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.432600 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" containerName="manager" Jul 10 16:02:52.436535 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.436511 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.452093 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.452062 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq"] Jul 10 16:02:52.458144 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.458112 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.459780 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.459758 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-86z89\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:52.461373 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.461341 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.471551 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.471524 2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" container="manager" Jul 10 16:02:52.491014 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.490980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ac57a2-e723-400c-9455-c11251aaacdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-796vc\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.491156 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.491020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxvh\" (UniqueName: \"kubernetes.io/projected/44ac57a2-e723-400c-9455-c11251aaacdd-kube-api-access-ktxvh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-796vc\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.494925 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.494889 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-86z89\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:52.496577 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.496545 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:52.499775 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.499750 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.593361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.592856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d9a128c-b2e5-4d67-bd07-334dc1358714-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-45mpq\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.593361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.592915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxvh\" (UniqueName: \"kubernetes.io/projected/44ac57a2-e723-400c-9455-c11251aaacdd-kube-api-access-ktxvh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-796vc\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.593361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.593043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ac57a2-e723-400c-9455-c11251aaacdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-796vc\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.593361 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.593071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxphl\" (UniqueName: \"kubernetes.io/projected/4d9a128c-b2e5-4d67-bd07-334dc1358714-kube-api-access-gxphl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-45mpq\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.593636 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.593521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ac57a2-e723-400c-9455-c11251aaacdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-796vc\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.612281 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.612193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxvh\" (UniqueName: \"kubernetes.io/projected/44ac57a2-e723-400c-9455-c11251aaacdd-kube-api-access-ktxvh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-796vc\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.665926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.665900 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.667933 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.667906 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-86z89\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:52.669394 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.669371 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:52.671810 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.671781 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.673552 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.673532 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:52.675397 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.675371 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:52.676821 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.676802 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-86z89\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:52.678462 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:52.678444 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:52.693879 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.693851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d9a128c-b2e5-4d67-bd07-334dc1358714-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-45mpq\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.694018 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.693944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxphl\" (UniqueName: \"kubernetes.io/projected/4d9a128c-b2e5-4d67-bd07-334dc1358714-kube-api-access-gxphl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-45mpq\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.694304 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.694282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d9a128c-b2e5-4d67-bd07-334dc1358714-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-45mpq\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.702946 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.702924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxphl\" (UniqueName: \"kubernetes.io/projected/4d9a128c-b2e5-4d67-bd07-334dc1358714-kube-api-access-gxphl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-45mpq\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.790971 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.790928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:52.795114 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.795092 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgpp9\" (UniqueName: \"kubernetes.io/projected/11341c1e-c842-4df5-93f0-2dc76dc605dd-kube-api-access-rgpp9\") pod \"11341c1e-c842-4df5-93f0-2dc76dc605dd\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " Jul 10 16:02:52.795180 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.795147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11341c1e-c842-4df5-93f0-2dc76dc605dd-extensions-socket-volume\") pod \"11341c1e-c842-4df5-93f0-2dc76dc605dd\" (UID: \"11341c1e-c842-4df5-93f0-2dc76dc605dd\") " Jul 10 16:02:52.795226 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.795212 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rcsn\" (UniqueName: \"kubernetes.io/projected/0141bc89-387e-4a08-ac8f-6fcb997bf471-kube-api-access-5rcsn\") pod \"0141bc89-387e-4a08-ac8f-6fcb997bf471\" (UID: \"0141bc89-387e-4a08-ac8f-6fcb997bf471\") " Jul 10 16:02:52.795655 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.795624 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11341c1e-c842-4df5-93f0-2dc76dc605dd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "11341c1e-c842-4df5-93f0-2dc76dc605dd" (UID: "11341c1e-c842-4df5-93f0-2dc76dc605dd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:02:52.797360 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.797329 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0141bc89-387e-4a08-ac8f-6fcb997bf471-kube-api-access-5rcsn" (OuterVolumeSpecName: "kube-api-access-5rcsn") pod "0141bc89-387e-4a08-ac8f-6fcb997bf471" (UID: "0141bc89-387e-4a08-ac8f-6fcb997bf471"). InnerVolumeSpecName "kube-api-access-5rcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:02:52.797360 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.797341 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11341c1e-c842-4df5-93f0-2dc76dc605dd-kube-api-access-rgpp9" (OuterVolumeSpecName: "kube-api-access-rgpp9") pod "11341c1e-c842-4df5-93f0-2dc76dc605dd" (UID: "11341c1e-c842-4df5-93f0-2dc76dc605dd"). InnerVolumeSpecName "kube-api-access-rgpp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:02:52.798353 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.798337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:52.895937 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.895902 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rcsn\" (UniqueName: \"kubernetes.io/projected/0141bc89-387e-4a08-ac8f-6fcb997bf471-kube-api-access-5rcsn\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:02:52.896074 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.895943 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgpp9\" (UniqueName: \"kubernetes.io/projected/11341c1e-c842-4df5-93f0-2dc76dc605dd-kube-api-access-rgpp9\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:02:52.896074 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.895959 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11341c1e-c842-4df5-93f0-2dc76dc605dd-extensions-socket-volume\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:02:52.940167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.940142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc"] Jul 10 16:02:52.943509 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:02:52.943475 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ac57a2_e723_400c_9455_c11251aaacdd.slice/crio-5920bcac21785c4678cee251e6114142fec0dbebf4d26e4e16db8c9b82203676 WatchSource:0}: Error finding container 5920bcac21785c4678cee251e6114142fec0dbebf4d26e4e16db8c9b82203676: Status 404 returned error can't find the container with id 5920bcac21785c4678cee251e6114142fec0dbebf4d26e4e16db8c9b82203676 Jul 10 16:02:52.973127 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:52.973024 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq"] Jul 10 16:02:52.980441 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:02:52.980416 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9a128c_b2e5_4d67_bd07_334dc1358714.slice/crio-5607697be302d32d5f2926abdbbe885d077f05a22bb34130ae7aef4494097ee2 WatchSource:0}: Error finding container 5607697be302d32d5f2926abdbbe885d077f05a22bb34130ae7aef4494097ee2: Status 404 returned error can't find the container with id 5607697be302d32d5f2926abdbbe885d077f05a22bb34130ae7aef4494097ee2 Jul 10 16:02:53.479821 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.479783 2576 generic.go:356] "Generic (PLEG): container finished" podID="11341c1e-c842-4df5-93f0-2dc76dc605dd" containerID="aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2" exitCode=0 Jul 10 16:02:53.479981 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.479895 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:53.479981 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.479938 2576 scope.go:117] "RemoveContainer" containerID="aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2" Jul 10 16:02:53.481737 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.481708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" event={"ID":"4d9a128c-b2e5-4d67-bd07-334dc1358714","Type":"ContainerStarted","Data":"ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb"} Jul 10 16:02:53.481864 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.481807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" event={"ID":"4d9a128c-b2e5-4d67-bd07-334dc1358714","Type":"ContainerStarted","Data":"5607697be302d32d5f2926abdbbe885d077f05a22bb34130ae7aef4494097ee2"} Jul 10 16:02:53.481864 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.481838 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:02:53.482532 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.482501 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:53.483149 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.483128 2576 generic.go:356] "Generic (PLEG): container finished" podID="0141bc89-387e-4a08-ac8f-6fcb997bf471" containerID="ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18" exitCode=0 Jul 10 16:02:53.483236 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.483177 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:53.485099 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.485037 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" podUID="ed735498-5602-4266-885c-ff13ab706fcc" containerName="manager" containerID="cri-o://71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1" gracePeriod=2 Jul 10 16:02:53.485192 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.485161 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-86z89\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:53.486902 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.486876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" event={"ID":"44ac57a2-e723-400c-9455-c11251aaacdd","Type":"ContainerStarted","Data":"f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d"} Jul 10 16:02:53.487012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.486925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" event={"ID":"44ac57a2-e723-400c-9455-c11251aaacdd","Type":"ContainerStarted","Data":"5920bcac21785c4678cee251e6114142fec0dbebf4d26e4e16db8c9b82203676"} Jul 10 16:02:53.487012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.486973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:02:53.487012 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.486983 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:53.488764 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.488736 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vkx7x\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vkx7x" Jul 10 16:02:53.490829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.490752 2576 scope.go:117] "RemoveContainer" containerID="aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2" Jul 10 16:02:53.491133 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.491111 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2\": container with ID starting with aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2 not found: ID does not exist" containerID="aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2" Jul 10 16:02:53.491218 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.491143 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2"} err="failed to get container status \"aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2\": rpc error: code = NotFound desc = could not find container \"aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2\": container with ID starting with aaace563b17bbbb2aa958663c873aa6d4e5f51ebd9a600943abb9229f06fa1c2 not found: ID does not exist" Jul 10 16:02:53.491218 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.491165 2576 scope.go:117] "RemoveContainer" containerID="ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18" Jul 10 16:02:53.497347 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.497320 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" path="/var/lib/kubelet/pods/0141bc89-387e-4a08-ac8f-6fcb997bf471/volumes" Jul 10 16:02:53.497669 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.497656 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11341c1e-c842-4df5-93f0-2dc76dc605dd" path="/var/lib/kubelet/pods/11341c1e-c842-4df5-93f0-2dc76dc605dd/volumes" Jul 10 16:02:53.500556 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.500534 2576 scope.go:117] "RemoveContainer" containerID="ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18" Jul 10 16:02:53.500956 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.500933 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18\": container with ID starting with ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18 not found: ID does not exist" containerID="ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18" Jul 10 16:02:53.501042 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.500961 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18"} err="failed to get container status \"ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18\": rpc error: code = NotFound desc = could not find container \"ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18\": container with ID starting with ec3d8ffe29d70bddf6087482a8a380bc8bf4a634ad93b839d8ac8094253d8d18 not found: ID does not exist" Jul 10 16:02:53.516675 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.516644 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-86z89\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="0141bc89-387e-4a08-ac8f-6fcb997bf471" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-86z89" Jul 10 16:02:53.517216 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.517175 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" podStartSLOduration=1.517162839 podStartE2EDuration="1.517162839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:02:53.514924776 +0000 UTC m=+462.585742845" watchObservedRunningTime="2026-07-10 16:02:53.517162839 +0000 UTC m=+462.587980909" Jul 10 16:02:53.518355 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:53.518333 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:53.547944 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.547892 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" podStartSLOduration=1.5478751769999999 podStartE2EDuration="1.547875177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:02:53.546020181 +0000 UTC m=+462.616838445" watchObservedRunningTime="2026-07-10 16:02:53.547875177 +0000 UTC m=+462.618693246" Jul 10 16:02:53.585513 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.585476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:53.686916 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.686881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:53.717719 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.717693 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:53.802469 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.802392 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed735498-5602-4266-885c-ff13ab706fcc-extensions-socket-volume\") pod \"ed735498-5602-4266-885c-ff13ab706fcc\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " Jul 10 16:02:53.802469 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.802430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrhpm\" (UniqueName: \"kubernetes.io/projected/ed735498-5602-4266-885c-ff13ab706fcc-kube-api-access-rrhpm\") pod \"ed735498-5602-4266-885c-ff13ab706fcc\" (UID: \"ed735498-5602-4266-885c-ff13ab706fcc\") " Jul 10 16:02:53.802782 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.802758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed735498-5602-4266-885c-ff13ab706fcc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ed735498-5602-4266-885c-ff13ab706fcc" (UID: "ed735498-5602-4266-885c-ff13ab706fcc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:02:53.804591 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.804572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed735498-5602-4266-885c-ff13ab706fcc-kube-api-access-rrhpm" (OuterVolumeSpecName: "kube-api-access-rrhpm") pod "ed735498-5602-4266-885c-ff13ab706fcc" (UID: "ed735498-5602-4266-885c-ff13ab706fcc"). InnerVolumeSpecName "kube-api-access-rrhpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:02:53.903219 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.903190 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed735498-5602-4266-885c-ff13ab706fcc-extensions-socket-volume\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:02:53.903362 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:53.903229 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhpm\" (UniqueName: \"kubernetes.io/projected/ed735498-5602-4266-885c-ff13ab706fcc-kube-api-access-rrhpm\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:02:54.495219 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:54.495185 2576 generic.go:356] "Generic (PLEG): container finished" podID="ed735498-5602-4266-885c-ff13ab706fcc" containerID="71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1" exitCode=0 Jul 10 16:02:54.495399 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:54.495222 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:54.495399 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:54.495299 2576 scope.go:117] "RemoveContainer" containerID="71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1" Jul 10 16:02:54.497536 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:54.497511 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:54.504278 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:54.504246 2576 scope.go:117] "RemoveContainer" containerID="71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1" Jul 10 16:02:54.504524 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:54.504508 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1\": container with ID starting with 71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1 not found: ID does not exist" containerID="71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1" Jul 10 16:02:54.504577 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:54.504529 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1"} err="failed to get container status \"71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1\": rpc error: code = NotFound desc = could not find container \"71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1\": container with ID starting with 71eca40ace8b6565000ab4d6ea9796a827bc67249db3402dbfdc10cdab2a09b1 not found: ID does not exist" Jul 10 16:02:54.504969 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:02:54.504951 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-g4tr9\" is forbidden: User \"system:node:ip-10-0-139-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-69.ec2.internal' and this object" podUID="ed735498-5602-4266-885c-ff13ab706fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g4tr9" Jul 10 16:02:55.495284 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:55.495233 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed735498-5602-4266-885c-ff13ab706fcc" path="/var/lib/kubelet/pods/ed735498-5602-4266-885c-ff13ab706fcc/volumes" Jul 10 16:02:58.366162 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:58.366129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:02:58.466320 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:02:58.466288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:01.950106 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:01.950068 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:02.050964 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:02.050929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:04.497808 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.497776 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:03:04.498204 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.497853 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:03:04.566798 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.566766 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq"] Jul 10 16:03:04.566980 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.566936 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" podUID="4d9a128c-b2e5-4d67-bd07-334dc1358714" containerName="manager" containerID="cri-o://ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb" gracePeriod=10 Jul 10 16:03:04.817083 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.817059 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:03:04.859430 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.859398 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh"] Jul 10 16:03:04.859911 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.859887 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9a128c-b2e5-4d67-bd07-334dc1358714" containerName="manager" Jul 10 16:03:04.859997 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.859914 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9a128c-b2e5-4d67-bd07-334dc1358714" containerName="manager" Jul 10 16:03:04.860034 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.860027 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9a128c-b2e5-4d67-bd07-334dc1358714" containerName="manager" Jul 10 16:03:04.862963 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.862941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:04.875814 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.875785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh"] Jul 10 16:03:04.893957 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.893929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p4sdh\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:04.894088 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.894006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96r4\" (UniqueName: \"kubernetes.io/projected/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-kube-api-access-c96r4\") pod \"kuadrant-operator-controller-manager-86878c769f-p4sdh\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:04.994478 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.994440 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxphl\" (UniqueName: \"kubernetes.io/projected/4d9a128c-b2e5-4d67-bd07-334dc1358714-kube-api-access-gxphl\") pod \"4d9a128c-b2e5-4d67-bd07-334dc1358714\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " Jul 10 16:03:04.994675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.994492 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d9a128c-b2e5-4d67-bd07-334dc1358714-extensions-socket-volume\") pod \"4d9a128c-b2e5-4d67-bd07-334dc1358714\" (UID: \"4d9a128c-b2e5-4d67-bd07-334dc1358714\") " Jul 10 16:03:04.994675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.994593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96r4\" (UniqueName: \"kubernetes.io/projected/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-kube-api-access-c96r4\") pod \"kuadrant-operator-controller-manager-86878c769f-p4sdh\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:04.994675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.994641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p4sdh\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:04.994929 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.994901 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a128c-b2e5-4d67-bd07-334dc1358714-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4d9a128c-b2e5-4d67-bd07-334dc1358714" (UID: "4d9a128c-b2e5-4d67-bd07-334dc1358714"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:03:04.994929 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.994921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p4sdh\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:04.996815 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:04.996793 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9a128c-b2e5-4d67-bd07-334dc1358714-kube-api-access-gxphl" (OuterVolumeSpecName: "kube-api-access-gxphl") pod "4d9a128c-b2e5-4d67-bd07-334dc1358714" (UID: "4d9a128c-b2e5-4d67-bd07-334dc1358714"). InnerVolumeSpecName "kube-api-access-gxphl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:03:05.003382 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.003361 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96r4\" (UniqueName: \"kubernetes.io/projected/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-kube-api-access-c96r4\") pod \"kuadrant-operator-controller-manager-86878c769f-p4sdh\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:05.095542 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.095449 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxphl\" (UniqueName: \"kubernetes.io/projected/4d9a128c-b2e5-4d67-bd07-334dc1358714-kube-api-access-gxphl\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:03:05.095542 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.095479 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d9a128c-b2e5-4d67-bd07-334dc1358714-extensions-socket-volume\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:03:05.173178 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.173143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:05.312588 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.312563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh"] Jul 10 16:03:05.314758 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:03:05.314729 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d764f0_f4a3_43dc_bf8b_fbd1a0f4c760.slice/crio-80f8e699d93562f56a3860a2552b9fb4505bbbc9e44fb7b2a3ce12a508b439e4 WatchSource:0}: Error finding container 80f8e699d93562f56a3860a2552b9fb4505bbbc9e44fb7b2a3ce12a508b439e4: Status 404 returned error can't find the container with id 80f8e699d93562f56a3860a2552b9fb4505bbbc9e44fb7b2a3ce12a508b439e4 Jul 10 16:03:05.552204 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.552167 2576 generic.go:356] "Generic (PLEG): container finished" podID="4d9a128c-b2e5-4d67-bd07-334dc1358714" containerID="ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb" exitCode=0 Jul 10 16:03:05.552675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.552208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" event={"ID":"4d9a128c-b2e5-4d67-bd07-334dc1358714","Type":"ContainerDied","Data":"ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb"} Jul 10 16:03:05.552675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.552288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" event={"ID":"4d9a128c-b2e5-4d67-bd07-334dc1358714","Type":"ContainerDied","Data":"5607697be302d32d5f2926abdbbe885d077f05a22bb34130ae7aef4494097ee2"} Jul 10 16:03:05.552675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.552217 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq" Jul 10 16:03:05.552675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.552314 2576 scope.go:117] "RemoveContainer" containerID="ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb" Jul 10 16:03:05.554291 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.554234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" event={"ID":"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760","Type":"ContainerStarted","Data":"d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429"} Jul 10 16:03:05.554857 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.554799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" event={"ID":"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760","Type":"ContainerStarted","Data":"80f8e699d93562f56a3860a2552b9fb4505bbbc9e44fb7b2a3ce12a508b439e4"} Jul 10 16:03:05.555088 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.555059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:05.567946 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.567843 2576 scope.go:117] "RemoveContainer" containerID="ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb" Jul 10 16:03:05.568158 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:03:05.568139 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb\": container with ID starting with ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb not found: ID does not exist" containerID="ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb" Jul 10 16:03:05.568203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.568166 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb"} err="failed to get container status \"ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb\": rpc error: code = NotFound desc = could not find container \"ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb\": container with ID starting with ea60a5036167cb6dc0cc0332017e62ad5a687f32b8d43af0f1731c26e96767fb not found: ID does not exist" Jul 10 16:03:05.579117 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.579064 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" podStartSLOduration=1.5790489829999999 podStartE2EDuration="1.579048983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:03:05.577713965 +0000 UTC m=+474.648532035" watchObservedRunningTime="2026-07-10 16:03:05.579048983 +0000 UTC m=+474.649867056" Jul 10 16:03:05.601565 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.601505 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq"] Jul 10 16:03:05.608374 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:05.608347 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-45mpq"] Jul 10 16:03:07.496000 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:07.495953 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9a128c-b2e5-4d67-bd07-334dc1358714" path="/var/lib/kubelet/pods/4d9a128c-b2e5-4d67-bd07-334dc1358714/volumes" Jul 10 16:03:08.601829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:08.601792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:08.703427 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:08.703383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:11.790656 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:11.790615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:11.898911 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:11.898877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:15.444501 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:15.444465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:16.331231 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.331196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:16.390669 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.390635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:16.563175 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.563145 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:03:16.612756 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.612682 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc"] Jul 10 16:03:16.612893 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.612869 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" podUID="44ac57a2-e723-400c-9455-c11251aaacdd" containerName="manager" containerID="cri-o://f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d" gracePeriod=10 Jul 10 16:03:16.860194 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.860171 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:03:16.901239 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.901165 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ac57a2-e723-400c-9455-c11251aaacdd-extensions-socket-volume\") pod \"44ac57a2-e723-400c-9455-c11251aaacdd\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " Jul 10 16:03:16.901239 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.901206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktxvh\" (UniqueName: \"kubernetes.io/projected/44ac57a2-e723-400c-9455-c11251aaacdd-kube-api-access-ktxvh\") pod \"44ac57a2-e723-400c-9455-c11251aaacdd\" (UID: \"44ac57a2-e723-400c-9455-c11251aaacdd\") " Jul 10 16:03:16.901660 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.901625 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac57a2-e723-400c-9455-c11251aaacdd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "44ac57a2-e723-400c-9455-c11251aaacdd" (UID: "44ac57a2-e723-400c-9455-c11251aaacdd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:03:16.903442 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:16.903421 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac57a2-e723-400c-9455-c11251aaacdd-kube-api-access-ktxvh" (OuterVolumeSpecName: "kube-api-access-ktxvh") pod "44ac57a2-e723-400c-9455-c11251aaacdd" (UID: "44ac57a2-e723-400c-9455-c11251aaacdd"). InnerVolumeSpecName "kube-api-access-ktxvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:03:17.002717 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.002680 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ac57a2-e723-400c-9455-c11251aaacdd-extensions-socket-volume\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:03:17.002717 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.002710 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktxvh\" (UniqueName: \"kubernetes.io/projected/44ac57a2-e723-400c-9455-c11251aaacdd-kube-api-access-ktxvh\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:03:17.404231 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.404195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:17.479827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.479784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:17.614502 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.614466 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ac57a2-e723-400c-9455-c11251aaacdd" containerID="f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d" exitCode=0 Jul 10 16:03:17.614880 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.614508 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" Jul 10 16:03:17.614880 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.614547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" event={"ID":"44ac57a2-e723-400c-9455-c11251aaacdd","Type":"ContainerDied","Data":"f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d"} Jul 10 16:03:17.614880 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.614596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc" event={"ID":"44ac57a2-e723-400c-9455-c11251aaacdd","Type":"ContainerDied","Data":"5920bcac21785c4678cee251e6114142fec0dbebf4d26e4e16db8c9b82203676"} Jul 10 16:03:17.614880 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.614620 2576 scope.go:117] "RemoveContainer" containerID="f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d" Jul 10 16:03:17.623573 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.623558 2576 scope.go:117] "RemoveContainer" containerID="f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d" Jul 10 16:03:17.623863 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:03:17.623840 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d\": container with ID starting with f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d not found: ID does not exist" containerID="f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d" Jul 10 16:03:17.623960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.623870 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d"} err="failed to get container status \"f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d\": rpc error: code = NotFound desc = could not find container \"f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d\": container with ID starting with f53f9ee36c2950c6c08e854ca77903ae64e163a84c7948b1be0c9e4ee69b137d not found: ID does not exist" Jul 10 16:03:17.634359 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.634339 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc"] Jul 10 16:03:17.638527 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:17.638509 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-796vc"] Jul 10 16:03:18.402331 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:18.402296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:18.466863 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:18.466830 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:19.494747 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:19.494708 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ac57a2-e723-400c-9455-c11251aaacdd" path="/var/lib/kubelet/pods/44ac57a2-e723-400c-9455-c11251aaacdd/volumes" Jul 10 16:03:21.115577 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.115547 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j"] Jul 10 16:03:21.115962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.115870 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ac57a2-e723-400c-9455-c11251aaacdd" containerName="manager" Jul 10 16:03:21.115962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.115881 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ac57a2-e723-400c-9455-c11251aaacdd" containerName="manager" Jul 10 16:03:21.115962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.115950 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ac57a2-e723-400c-9455-c11251aaacdd" containerName="manager" Jul 10 16:03:21.118773 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.118753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.121995 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.121979 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zjbhv\"" Jul 10 16:03:21.135412 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.135393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j"] Jul 10 16:03:21.148269 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.148226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:21.240706 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.240706 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt7ct\" (UniqueName: \"kubernetes.io/projected/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-kube-api-access-xt7ct\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.240916 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.240916 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.240916 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.241008 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.241008 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.241008 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.241008 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.240998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343713 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt7ct\" (UniqueName: \"kubernetes.io/projected/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-kube-api-access-xt7ct\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.343918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.344241 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.343949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.345317 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.345286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.345502 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.345476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.345667 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.345545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.345811 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.345675 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.345934 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.345742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.346977 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.346949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.347816 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.347799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.354546 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.354526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.354781 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.354762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt7ct\" (UniqueName: \"kubernetes.io/projected/7a6c8ffd-3477-4bd1-8416-a12ff141e3af-kube-api-access-xt7ct\") pod \"maas-default-gateway-openshift-default-56654dcf7-q7p9j\" (UID: \"7a6c8ffd-3477-4bd1-8416-a12ff141e3af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.428962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.428859 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:21.575064 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.575040 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j"] Jul 10 16:03:21.576880 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:03:21.576858 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6c8ffd_3477_4bd1_8416_a12ff141e3af.slice/crio-c94284fa7f2df844d69bd32d74547de2f6cbd3f81f735c6bf7f3bd389b04a47d WatchSource:0}: Error finding container c94284fa7f2df844d69bd32d74547de2f6cbd3f81f735c6bf7f3bd389b04a47d: Status 404 returned error can't find the container with id c94284fa7f2df844d69bd32d74547de2f6cbd3f81f735c6bf7f3bd389b04a47d Jul 10 16:03:21.597932 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.597901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:21.634499 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:21.634466 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" event={"ID":"7a6c8ffd-3477-4bd1-8416-a12ff141e3af","Type":"ContainerStarted","Data":"c94284fa7f2df844d69bd32d74547de2f6cbd3f81f735c6bf7f3bd389b04a47d"} Jul 10 16:03:22.600389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:22.600351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:22.700979 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:22.700946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:24.756076 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:24.756038 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:03:24.756341 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:24.756124 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:03:24.756341 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:24.756159 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:03:25.655408 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:25.655370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" event={"ID":"7a6c8ffd-3477-4bd1-8416-a12ff141e3af","Type":"ContainerStarted","Data":"e171a24e0149b9fe903cedf6b3999c7e4c3acea64ad719400e1a2b5d547ad35f"} Jul 10 16:03:25.679003 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:25.678949 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" podStartSLOduration=1.502166576 podStartE2EDuration="4.678930422s" totalImagesPullingTime="3.176763846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:03:25.676820863 +0000 UTC m=+494.747638933" watchObservedRunningTime="2026-07-10 16:03:25.678930422 +0000 UTC m=+494.749748494" Jul 10 16:03:26.429874 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:26.429834 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:26.434506 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:26.434479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:26.659601 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:26.659566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:26.660615 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:26.660594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:03:27.687901 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:27.687868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:27.792305 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:27.792246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:27.892454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:27.892419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:35.220609 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.220566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:35.227452 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.227426 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:35.229738 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.229720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.231960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.231936 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 16:03:35.232069 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.231936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jlgqs\"" Jul 10 16:03:35.241520 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.241493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:35.324602 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.324571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:35.335676 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.335642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:35.362646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.362613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbbr\" (UniqueName: \"kubernetes.io/projected/dba7b53a-4522-4066-99c1-50353e8ab856-kube-api-access-dfbbr\") pod \"limitador-limitador-64c94fd64f-dmnjb\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.362797 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.362653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dba7b53a-4522-4066-99c1-50353e8ab856-config-file\") pod \"limitador-limitador-64c94fd64f-dmnjb\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.398620 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.398586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:35.463312 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.463277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbbr\" (UniqueName: \"kubernetes.io/projected/dba7b53a-4522-4066-99c1-50353e8ab856-kube-api-access-dfbbr\") pod \"limitador-limitador-64c94fd64f-dmnjb\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.463474 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.463329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dba7b53a-4522-4066-99c1-50353e8ab856-config-file\") pod \"limitador-limitador-64c94fd64f-dmnjb\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.463884 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.463865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dba7b53a-4522-4066-99c1-50353e8ab856-config-file\") pod \"limitador-limitador-64c94fd64f-dmnjb\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.475599 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.475529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbbr\" (UniqueName: \"kubernetes.io/projected/dba7b53a-4522-4066-99c1-50353e8ab856-kube-api-access-dfbbr\") pod \"limitador-limitador-64c94fd64f-dmnjb\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.543289 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.543230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:35.684649 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.684601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:35.686821 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:03:35.686793 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba7b53a_4522_4066_99c1_50353e8ab856.slice/crio-9c8e035d2d57f60914bd0e8744a3eb4cf3830660d6fb91a715e42b30548f7fd1 WatchSource:0}: Error finding container 9c8e035d2d57f60914bd0e8744a3eb4cf3830660d6fb91a715e42b30548f7fd1: Status 404 returned error can't find the container with id 9c8e035d2d57f60914bd0e8744a3eb4cf3830660d6fb91a715e42b30548f7fd1 Jul 10 16:03:35.705215 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:35.705188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" event={"ID":"dba7b53a-4522-4066-99c1-50353e8ab856","Type":"ContainerStarted","Data":"9c8e035d2d57f60914bd0e8744a3eb4cf3830660d6fb91a715e42b30548f7fd1"} Jul 10 16:03:36.069446 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:36.069413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:36.128718 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:36.128683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:38.721917 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:38.721814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" event={"ID":"dba7b53a-4522-4066-99c1-50353e8ab856","Type":"ContainerStarted","Data":"29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d"} Jul 10 16:03:38.722304 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:38.721946 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:38.740424 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:38.740371 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" podStartSLOduration=0.980946794 podStartE2EDuration="3.740355698s" totalImagesPullingTime="2.759408904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:03:38.739367824 +0000 UTC m=+507.810185911" watchObservedRunningTime="2026-07-10 16:03:38.740355698 +0000 UTC m=+507.811173767" Jul 10 16:03:49.727667 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:49.727623 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:51.643472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:51.643433 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:51.643864 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:51.643631 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" podUID="dba7b53a-4522-4066-99c1-50353e8ab856" containerName="limitador" containerID="cri-o://29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d" gracePeriod=30 Jul 10 16:03:52.193901 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.193871 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:52.314613 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.314504 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dba7b53a-4522-4066-99c1-50353e8ab856-config-file\") pod \"dba7b53a-4522-4066-99c1-50353e8ab856\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " Jul 10 16:03:52.314613 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.314570 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfbbr\" (UniqueName: \"kubernetes.io/projected/dba7b53a-4522-4066-99c1-50353e8ab856-kube-api-access-dfbbr\") pod \"dba7b53a-4522-4066-99c1-50353e8ab856\" (UID: \"dba7b53a-4522-4066-99c1-50353e8ab856\") " Jul 10 16:03:52.314977 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.314949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba7b53a-4522-4066-99c1-50353e8ab856-config-file" (OuterVolumeSpecName: "config-file") pod "dba7b53a-4522-4066-99c1-50353e8ab856" (UID: "dba7b53a-4522-4066-99c1-50353e8ab856"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:03:52.316842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.316818 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba7b53a-4522-4066-99c1-50353e8ab856-kube-api-access-dfbbr" (OuterVolumeSpecName: "kube-api-access-dfbbr") pod "dba7b53a-4522-4066-99c1-50353e8ab856" (UID: "dba7b53a-4522-4066-99c1-50353e8ab856"). InnerVolumeSpecName "kube-api-access-dfbbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:03:52.415873 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.415838 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dba7b53a-4522-4066-99c1-50353e8ab856-config-file\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:03:52.415873 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.415867 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbbr\" (UniqueName: \"kubernetes.io/projected/dba7b53a-4522-4066-99c1-50353e8ab856-kube-api-access-dfbbr\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:03:52.789451 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.789416 2576 generic.go:356] "Generic (PLEG): container finished" podID="dba7b53a-4522-4066-99c1-50353e8ab856" containerID="29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d" exitCode=0 Jul 10 16:03:52.789947 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.789464 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" Jul 10 16:03:52.789947 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.789508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" event={"ID":"dba7b53a-4522-4066-99c1-50353e8ab856","Type":"ContainerDied","Data":"29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d"} Jul 10 16:03:52.789947 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.789575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dmnjb" event={"ID":"dba7b53a-4522-4066-99c1-50353e8ab856","Type":"ContainerDied","Data":"9c8e035d2d57f60914bd0e8744a3eb4cf3830660d6fb91a715e42b30548f7fd1"} Jul 10 16:03:52.789947 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.789613 2576 scope.go:117] "RemoveContainer" containerID="29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d" Jul 10 16:03:52.798297 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.798272 2576 scope.go:117] "RemoveContainer" containerID="29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d" Jul 10 16:03:52.798558 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:03:52.798542 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d\": container with ID starting with 29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d not found: ID does not exist" containerID="29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d" Jul 10 16:03:52.798612 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.798566 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d"} err="failed to get container status \"29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d\": rpc error: code = NotFound desc = could not find container \"29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d\": container with ID starting with 29500a1e6da6994d7a4b9270e35e53a6381c2c1d2896eb653e70258ac80d392d not found: ID does not exist" Jul 10 16:03:52.811111 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.811079 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:52.816510 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:52.816481 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dmnjb"] Jul 10 16:03:53.494635 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.494597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:53.495109 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.495059 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba7b53a-4522-4066-99c1-50353e8ab856" path="/var/lib/kubelet/pods/dba7b53a-4522-4066-99c1-50353e8ab856/volumes" Jul 10 16:03:53.667942 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.667909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:53.965472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.965437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6"] Jul 10 16:03:53.966050 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.966015 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba7b53a-4522-4066-99c1-50353e8ab856" containerName="limitador" Jul 10 16:03:53.966050 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.966042 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba7b53a-4522-4066-99c1-50353e8ab856" containerName="limitador" Jul 10 16:03:53.966286 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.966198 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba7b53a-4522-4066-99c1-50353e8ab856" containerName="limitador" Jul 10 16:03:53.969461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.969442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:53.971699 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.971673 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 16:03:53.971809 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.971719 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 16:03:53.972436 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.972413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ddjdp\"" Jul 10 16:03:53.972519 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.972442 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 16:03:53.981360 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:53.981337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6"] Jul 10 16:03:54.011777 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.011742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:54.132038 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.132001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74fa0c47-0dcc-444e-9b6a-532887254c61-data\") pod \"postgres-69fb4594cf-dk5f6\" (UID: \"74fa0c47-0dcc-444e-9b6a-532887254c61\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.132207 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.132045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszlh\" (UniqueName: \"kubernetes.io/projected/74fa0c47-0dcc-444e-9b6a-532887254c61-kube-api-access-gszlh\") pod \"postgres-69fb4594cf-dk5f6\" (UID: \"74fa0c47-0dcc-444e-9b6a-532887254c61\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.233122 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.233028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74fa0c47-0dcc-444e-9b6a-532887254c61-data\") pod \"postgres-69fb4594cf-dk5f6\" (UID: \"74fa0c47-0dcc-444e-9b6a-532887254c61\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.233122 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.233081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszlh\" (UniqueName: \"kubernetes.io/projected/74fa0c47-0dcc-444e-9b6a-532887254c61-kube-api-access-gszlh\") pod \"postgres-69fb4594cf-dk5f6\" (UID: \"74fa0c47-0dcc-444e-9b6a-532887254c61\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.233510 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.233490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74fa0c47-0dcc-444e-9b6a-532887254c61-data\") pod \"postgres-69fb4594cf-dk5f6\" (UID: \"74fa0c47-0dcc-444e-9b6a-532887254c61\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.242076 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.242053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszlh\" (UniqueName: \"kubernetes.io/projected/74fa0c47-0dcc-444e-9b6a-532887254c61-kube-api-access-gszlh\") pod \"postgres-69fb4594cf-dk5f6\" (UID: \"74fa0c47-0dcc-444e-9b6a-532887254c61\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.281015 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.280978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:03:54.421330 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.421305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6"] Jul 10 16:03:54.423507 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:03:54.423480 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fa0c47_0dcc_444e_9b6a_532887254c61.slice/crio-0d2a8f58201206994ca8fa3c146ce2a79c5353189247d4a1f706317ce5ff418a WatchSource:0}: Error finding container 0d2a8f58201206994ca8fa3c146ce2a79c5353189247d4a1f706317ce5ff418a: Status 404 returned error can't find the container with id 0d2a8f58201206994ca8fa3c146ce2a79c5353189247d4a1f706317ce5ff418a Jul 10 16:03:54.800379 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.800345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" event={"ID":"74fa0c47-0dcc-444e-9b6a-532887254c61","Type":"ContainerStarted","Data":"0d2a8f58201206994ca8fa3c146ce2a79c5353189247d4a1f706317ce5ff418a"} Jul 10 16:03:54.940902 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:54.940869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:03:55.003699 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:03:55.003664 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:00.834416 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:00.834386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" event={"ID":"74fa0c47-0dcc-444e-9b6a-532887254c61","Type":"ContainerStarted","Data":"2e336173651e7b6c390c2a536575ef79c13455d545659cc7ddf5db32eb2da0f5"} Jul 10 16:04:00.834915 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:00.834446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:04:00.857055 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:00.856992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" podStartSLOduration=2.311083698 podStartE2EDuration="7.856971567s" totalImagesPullingTime="5.545887869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:04:00.856073994 +0000 UTC m=+529.926892064" watchObservedRunningTime="2026-07-10 16:04:00.856971567 +0000 UTC m=+529.927789639" Jul 10 16:04:06.869769 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:06.869685 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dk5f6" Jul 10 16:04:07.255213 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:07.255175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:07.442223 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:07.442188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:14.236917 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.236878 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-db579b5df-lsgjf"] Jul 10 16:04:14.246857 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.246823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.249147 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.249099 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 16:04:14.249345 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.249099 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d77qb\"" Jul 10 16:04:14.249345 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.249218 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-db579b5df-lsgjf"] Jul 10 16:04:14.249345 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.249099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 16:04:14.250099 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.250075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:14.307713 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.307676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b9dd8e-6beb-4caf-a283-826404088eca-webhook-cert\") pod \"maas-controller-db579b5df-lsgjf\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.307891 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.307726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56x8x\" (UniqueName: \"kubernetes.io/projected/b4b9dd8e-6beb-4caf-a283-826404088eca-kube-api-access-56x8x\") pod \"maas-controller-db579b5df-lsgjf\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.348798 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.348741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:14.408740 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.408700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56x8x\" (UniqueName: \"kubernetes.io/projected/b4b9dd8e-6beb-4caf-a283-826404088eca-kube-api-access-56x8x\") pod \"maas-controller-db579b5df-lsgjf\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.408939 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.408807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b9dd8e-6beb-4caf-a283-826404088eca-webhook-cert\") pod \"maas-controller-db579b5df-lsgjf\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.411441 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.411414 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787666694b-hnvgr"] Jul 10 16:04:14.411645 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.411624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b9dd8e-6beb-4caf-a283-826404088eca-webhook-cert\") pod \"maas-controller-db579b5df-lsgjf\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.419547 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.419520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56x8x\" (UniqueName: \"kubernetes.io/projected/b4b9dd8e-6beb-4caf-a283-826404088eca-kube-api-access-56x8x\") pod \"maas-controller-db579b5df-lsgjf\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.420030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.420015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.426271 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.426222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787666694b-hnvgr"] Jul 10 16:04:14.432629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.432594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:14.510313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.510206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-webhook-cert\") pod \"maas-controller-787666694b-hnvgr\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.510313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.510275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngt4g\" (UniqueName: \"kubernetes.io/projected/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-kube-api-access-ngt4g\") pod \"maas-controller-787666694b-hnvgr\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.558391 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.558357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:14.611192 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.611166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-webhook-cert\") pod \"maas-controller-787666694b-hnvgr\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.611446 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.611425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngt4g\" (UniqueName: \"kubernetes.io/projected/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-kube-api-access-ngt4g\") pod \"maas-controller-787666694b-hnvgr\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.614122 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.614090 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-webhook-cert\") pod \"maas-controller-787666694b-hnvgr\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.619722 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.619586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngt4g\" (UniqueName: \"kubernetes.io/projected/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-kube-api-access-ngt4g\") pod \"maas-controller-787666694b-hnvgr\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.697146 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.697118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-db579b5df-lsgjf"] Jul 10 16:04:14.699462 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:04:14.699437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b9dd8e_6beb_4caf_a283_826404088eca.slice/crio-f447528403b0d390ea2054516ee3b344a5f970ee1582776e2874d213a3d1c182 WatchSource:0}: Error finding container f447528403b0d390ea2054516ee3b344a5f970ee1582776e2874d213a3d1c182: Status 404 returned error can't find the container with id f447528403b0d390ea2054516ee3b344a5f970ee1582776e2874d213a3d1c182 Jul 10 16:04:14.738110 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.738072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:14.901894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.901774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-db579b5df-lsgjf" event={"ID":"b4b9dd8e-6beb-4caf-a283-826404088eca","Type":"ContainerStarted","Data":"f447528403b0d390ea2054516ee3b344a5f970ee1582776e2874d213a3d1c182"} Jul 10 16:04:14.903935 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:14.903914 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787666694b-hnvgr"] Jul 10 16:04:14.905795 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:04:14.905765 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9249bc9e_2486_4e3f_ad63_9312dbd4d13c.slice/crio-bbc429c1dc589be89c08f970b8de44ff20b05ff37e706d2756d0e5fd0840cd00 WatchSource:0}: Error finding container bbc429c1dc589be89c08f970b8de44ff20b05ff37e706d2756d0e5fd0840cd00: Status 404 returned error can't find the container with id bbc429c1dc589be89c08f970b8de44ff20b05ff37e706d2756d0e5fd0840cd00 Jul 10 16:04:15.042342 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:15.042212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:15.910766 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:15.910725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787666694b-hnvgr" event={"ID":"9249bc9e-2486-4e3f-ad63-9312dbd4d13c","Type":"ContainerStarted","Data":"bbc429c1dc589be89c08f970b8de44ff20b05ff37e706d2756d0e5fd0840cd00"} Jul 10 16:04:17.766756 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.766720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:17.867978 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.867857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:17.921892 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.921842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-db579b5df-lsgjf" event={"ID":"b4b9dd8e-6beb-4caf-a283-826404088eca","Type":"ContainerStarted","Data":"39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99"} Jul 10 16:04:17.922100 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.921953 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:17.923517 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.923492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787666694b-hnvgr" event={"ID":"9249bc9e-2486-4e3f-ad63-9312dbd4d13c","Type":"ContainerStarted","Data":"e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf"} Jul 10 16:04:17.923643 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.923559 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:17.941067 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.940991 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-db579b5df-lsgjf" podStartSLOduration=1.102257611 podStartE2EDuration="3.940973703s" totalImagesPullingTime="2.838716092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:04:17.939484728 +0000 UTC m=+547.010302796" watchObservedRunningTime="2026-07-10 16:04:17.940973703 +0000 UTC m=+547.011791773" Jul 10 16:04:17.959624 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:17.959570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787666694b-hnvgr" podStartSLOduration=1.315928451 podStartE2EDuration="3.959553296s" totalImagesPullingTime="2.643624845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:04:17.956921655 +0000 UTC m=+547.027739725" watchObservedRunningTime="2026-07-10 16:04:17.959553296 +0000 UTC m=+547.030371364" Jul 10 16:04:28.935336 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:28.935301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:04:28.935839 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:28.935612 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:29.001167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.001124 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-db579b5df-lsgjf"] Jul 10 16:04:29.001377 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.001354 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-db579b5df-lsgjf" podUID="b4b9dd8e-6beb-4caf-a283-826404088eca" containerName="manager" containerID="cri-o://39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99" gracePeriod=30 Jul 10 16:04:29.255526 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.255496 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:29.348049 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.348010 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b9dd8e-6beb-4caf-a283-826404088eca-webhook-cert\") pod \"b4b9dd8e-6beb-4caf-a283-826404088eca\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " Jul 10 16:04:29.348203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.348146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56x8x\" (UniqueName: \"kubernetes.io/projected/b4b9dd8e-6beb-4caf-a283-826404088eca-kube-api-access-56x8x\") pod \"b4b9dd8e-6beb-4caf-a283-826404088eca\" (UID: \"b4b9dd8e-6beb-4caf-a283-826404088eca\") " Jul 10 16:04:29.350483 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.350449 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b9dd8e-6beb-4caf-a283-826404088eca-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b4b9dd8e-6beb-4caf-a283-826404088eca" (UID: "b4b9dd8e-6beb-4caf-a283-826404088eca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:04:29.350604 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.350539 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b9dd8e-6beb-4caf-a283-826404088eca-kube-api-access-56x8x" (OuterVolumeSpecName: "kube-api-access-56x8x") pod "b4b9dd8e-6beb-4caf-a283-826404088eca" (UID: "b4b9dd8e-6beb-4caf-a283-826404088eca"). InnerVolumeSpecName "kube-api-access-56x8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:04:29.449363 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.449320 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56x8x\" (UniqueName: \"kubernetes.io/projected/b4b9dd8e-6beb-4caf-a283-826404088eca-kube-api-access-56x8x\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:04:29.449363 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.449356 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b9dd8e-6beb-4caf-a283-826404088eca-webhook-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:04:29.982414 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.982379 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4b9dd8e-6beb-4caf-a283-826404088eca" containerID="39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99" exitCode=0 Jul 10 16:04:29.982813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.982427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-db579b5df-lsgjf" Jul 10 16:04:29.982813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.982434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-db579b5df-lsgjf" event={"ID":"b4b9dd8e-6beb-4caf-a283-826404088eca","Type":"ContainerDied","Data":"39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99"} Jul 10 16:04:29.982813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.982475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-db579b5df-lsgjf" event={"ID":"b4b9dd8e-6beb-4caf-a283-826404088eca","Type":"ContainerDied","Data":"f447528403b0d390ea2054516ee3b344a5f970ee1582776e2874d213a3d1c182"} Jul 10 16:04:29.982813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.982496 2576 scope.go:117] "RemoveContainer" containerID="39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99" Jul 10 16:04:29.991444 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.991426 2576 scope.go:117] "RemoveContainer" containerID="39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99" Jul 10 16:04:29.991696 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:04:29.991680 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99\": container with ID starting with 39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99 not found: ID does not exist" containerID="39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99" Jul 10 16:04:29.991737 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.991705 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99"} err="failed to get container status \"39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99\": rpc error: code = NotFound desc = could not find container \"39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99\": container with ID starting with 39124262d20f0061f4a4caa9485034f70aa3ab1cce0879d1571dd3c2176acb99 not found: ID does not exist" Jul 10 16:04:29.999565 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:29.999537 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-db579b5df-lsgjf"] Jul 10 16:04:30.004037 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:30.004014 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-db579b5df-lsgjf"] Jul 10 16:04:31.501328 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:31.501293 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b9dd8e-6beb-4caf-a283-826404088eca" path="/var/lib/kubelet/pods/b4b9dd8e-6beb-4caf-a283-826404088eca/volumes" Jul 10 16:04:52.962827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:52.962794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:53.506832 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.506794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:53.647900 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.647872 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k"] Jul 10 16:04:53.648237 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.648225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b9dd8e-6beb-4caf-a283-826404088eca" containerName="manager" Jul 10 16:04:53.648310 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.648239 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b9dd8e-6beb-4caf-a283-826404088eca" containerName="manager" Jul 10 16:04:53.648359 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.648311 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b9dd8e-6beb-4caf-a283-826404088eca" containerName="manager" Jul 10 16:04:53.653512 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.653496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.656568 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.656548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d269e-1902-44a7-9c7b-96efca165806-maas-api-tls\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.656645 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.656579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d269e-1902-44a7-9c7b-96efca165806-openshift-service-ca\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.656645 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.656622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmq6\" (UniqueName: \"kubernetes.io/projected/069d269e-1902-44a7-9c7b-96efca165806-kube-api-access-lzmq6\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.691313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.691283 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 16:04:53.691463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.691291 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xbv8r\"" Jul 10 16:04:53.692944 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.692926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k"] Jul 10 16:04:53.706058 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.706016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:53.757713 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.757627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d269e-1902-44a7-9c7b-96efca165806-maas-api-tls\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.757713 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.757667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d269e-1902-44a7-9c7b-96efca165806-openshift-service-ca\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.757903 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.757726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmq6\" (UniqueName: \"kubernetes.io/projected/069d269e-1902-44a7-9c7b-96efca165806-kube-api-access-lzmq6\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.758481 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.758396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d269e-1902-44a7-9c7b-96efca165806-openshift-service-ca\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.760321 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.760297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d269e-1902-44a7-9c7b-96efca165806-maas-api-tls\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.771276 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.771232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmq6\" (UniqueName: \"kubernetes.io/projected/069d269e-1902-44a7-9c7b-96efca165806-kube-api-access-lzmq6\") pod \"maas-api-6c4df6cbf-bdw4k\" (UID: \"069d269e-1902-44a7-9c7b-96efca165806\") " pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:53.963401 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:53.963367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:54.114325 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.114288 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k"] Jul 10 16:04:54.117901 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:04:54.117873 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069d269e_1902_44a7_9c7b_96efca165806.slice/crio-b9b74a4bc5c1a5a81fc833e9f68e50eed0fc4a940fd972a5aaa549f9f993b543 WatchSource:0}: Error finding container b9b74a4bc5c1a5a81fc833e9f68e50eed0fc4a940fd972a5aaa549f9f993b543: Status 404 returned error can't find the container with id b9b74a4bc5c1a5a81fc833e9f68e50eed0fc4a940fd972a5aaa549f9f993b543 Jul 10 16:04:54.314091 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.314005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:54.414995 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.414965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:04:54.475787 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.475749 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-9v2vw"] Jul 10 16:04:54.480920 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.480896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.483888 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.483863 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qh84z\"" Jul 10 16:04:54.484350 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.484325 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 16:04:54.493324 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.493301 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-fq6hb"] Jul 10 16:04:54.497125 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.497089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-9v2vw"] Jul 10 16:04:54.497245 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.497209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.517334 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.517304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-fq6hb"] Jul 10 16:04:54.563902 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.563869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e3f4b2-f74d-43d6-b21c-48d250837725-plugins-config-volume\") pod \"payload-processing-58f86c7798-9v2vw\" (UID: \"f6e3f4b2-f74d-43d6-b21c-48d250837725\") " pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.564089 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.563942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhz86\" (UniqueName: \"kubernetes.io/projected/f6e3f4b2-f74d-43d6-b21c-48d250837725-kube-api-access-nhz86\") pod \"payload-processing-58f86c7798-9v2vw\" (UID: \"f6e3f4b2-f74d-43d6-b21c-48d250837725\") " pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.564089 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.563969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cd78d6-3169-425c-b3c1-7454f496c253-plugins-config-volume\") pod \"payload-pre-processing-7964457696-fq6hb\" (UID: \"51cd78d6-3169-425c-b3c1-7454f496c253\") " pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.564089 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.563989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrrj\" (UniqueName: \"kubernetes.io/projected/51cd78d6-3169-425c-b3c1-7454f496c253-kube-api-access-gzrrj\") pod \"payload-pre-processing-7964457696-fq6hb\" (UID: \"51cd78d6-3169-425c-b3c1-7454f496c253\") " pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.664988 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.664908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cd78d6-3169-425c-b3c1-7454f496c253-plugins-config-volume\") pod \"payload-pre-processing-7964457696-fq6hb\" (UID: \"51cd78d6-3169-425c-b3c1-7454f496c253\") " pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.664988 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.664946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrrj\" (UniqueName: \"kubernetes.io/projected/51cd78d6-3169-425c-b3c1-7454f496c253-kube-api-access-gzrrj\") pod \"payload-pre-processing-7964457696-fq6hb\" (UID: \"51cd78d6-3169-425c-b3c1-7454f496c253\") " pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.664988 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.664987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e3f4b2-f74d-43d6-b21c-48d250837725-plugins-config-volume\") pod \"payload-processing-58f86c7798-9v2vw\" (UID: \"f6e3f4b2-f74d-43d6-b21c-48d250837725\") " pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.665198 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.665087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhz86\" (UniqueName: \"kubernetes.io/projected/f6e3f4b2-f74d-43d6-b21c-48d250837725-kube-api-access-nhz86\") pod \"payload-processing-58f86c7798-9v2vw\" (UID: \"f6e3f4b2-f74d-43d6-b21c-48d250837725\") " pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.665678 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.665660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cd78d6-3169-425c-b3c1-7454f496c253-plugins-config-volume\") pod \"payload-pre-processing-7964457696-fq6hb\" (UID: \"51cd78d6-3169-425c-b3c1-7454f496c253\") " pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.665869 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.665849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e3f4b2-f74d-43d6-b21c-48d250837725-plugins-config-volume\") pod \"payload-processing-58f86c7798-9v2vw\" (UID: \"f6e3f4b2-f74d-43d6-b21c-48d250837725\") " pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.673601 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.673575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrrj\" (UniqueName: \"kubernetes.io/projected/51cd78d6-3169-425c-b3c1-7454f496c253-kube-api-access-gzrrj\") pod \"payload-pre-processing-7964457696-fq6hb\" (UID: \"51cd78d6-3169-425c-b3c1-7454f496c253\") " pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.674028 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.674006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhz86\" (UniqueName: \"kubernetes.io/projected/f6e3f4b2-f74d-43d6-b21c-48d250837725-kube-api-access-nhz86\") pod \"payload-processing-58f86c7798-9v2vw\" (UID: \"f6e3f4b2-f74d-43d6-b21c-48d250837725\") " pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.792575 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.792537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:54.808534 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.808507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:54.964425 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.964361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-9v2vw"] Jul 10 16:04:54.966869 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:04:54.966826 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e3f4b2_f74d_43d6_b21c_48d250837725.slice/crio-df6a8385a38ca6085a58409fe01cd01d3213f3447eec9b1e26a4389a2ec3a5e4 WatchSource:0}: Error finding container df6a8385a38ca6085a58409fe01cd01d3213f3447eec9b1e26a4389a2ec3a5e4: Status 404 returned error can't find the container with id df6a8385a38ca6085a58409fe01cd01d3213f3447eec9b1e26a4389a2ec3a5e4 Jul 10 16:04:54.988910 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:54.988885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-fq6hb"] Jul 10 16:04:54.990627 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:04:54.990600 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cd78d6_3169_425c_b3c1_7454f496c253.slice/crio-6b7b95fc965ae1068e7c99e3596f6cbe24b1b2b5287ad75c422a0b4a58ea3858 WatchSource:0}: Error finding container 6b7b95fc965ae1068e7c99e3596f6cbe24b1b2b5287ad75c422a0b4a58ea3858: Status 404 returned error can't find the container with id 6b7b95fc965ae1068e7c99e3596f6cbe24b1b2b5287ad75c422a0b4a58ea3858 Jul 10 16:04:55.115421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:55.115380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" event={"ID":"51cd78d6-3169-425c-b3c1-7454f496c253","Type":"ContainerStarted","Data":"6b7b95fc965ae1068e7c99e3596f6cbe24b1b2b5287ad75c422a0b4a58ea3858"} Jul 10 16:04:55.116632 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:55.116605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" event={"ID":"069d269e-1902-44a7-9c7b-96efca165806","Type":"ContainerStarted","Data":"b9b74a4bc5c1a5a81fc833e9f68e50eed0fc4a940fd972a5aaa549f9f993b543"} Jul 10 16:04:55.117585 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:55.117564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" event={"ID":"f6e3f4b2-f74d-43d6-b21c-48d250837725","Type":"ContainerStarted","Data":"df6a8385a38ca6085a58409fe01cd01d3213f3447eec9b1e26a4389a2ec3a5e4"} Jul 10 16:04:57.134225 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.134150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" event={"ID":"f6e3f4b2-f74d-43d6-b21c-48d250837725","Type":"ContainerStarted","Data":"dcad4e68e93b23fa3d0299351edc93c74d9c8fc7cde7bff805621ec801dfb210"} Jul 10 16:04:57.134584 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.134395 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:04:57.135854 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.135830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" event={"ID":"51cd78d6-3169-425c-b3c1-7454f496c253","Type":"ContainerStarted","Data":"0314f29dd0a18f8e68fda06557321b46c75644263dff21f97fd2ba46b971a330"} Jul 10 16:04:57.135951 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.135883 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:04:57.137303 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.137278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" event={"ID":"069d269e-1902-44a7-9c7b-96efca165806","Type":"ContainerStarted","Data":"186781f69040fab6d74770a866e50921a0936adf7962f4f1e6afa07bcbe3b77b"} Jul 10 16:04:57.137412 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.137390 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:04:57.156550 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.156496 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" podStartSLOduration=1.375210675 podStartE2EDuration="3.156480434s" totalImagesPullingTime="1.781269759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:04:57.154201297 +0000 UTC m=+586.225019367" watchObservedRunningTime="2026-07-10 16:04:57.156480434 +0000 UTC m=+586.227298514" Jul 10 16:04:57.198227 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.198155 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" podStartSLOduration=1.442063259 podStartE2EDuration="3.19813029s" totalImagesPullingTime="1.756067031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:04:57.175651833 +0000 UTC m=+586.246469912" watchObservedRunningTime="2026-07-10 16:04:57.19813029 +0000 UTC m=+586.268948358" Jul 10 16:04:57.199074 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:04:57.199019 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" podStartSLOduration=1.5694327829999999 podStartE2EDuration="4.199006916s" totalImagesPullingTime="2.629574133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:04:57.195925602 +0000 UTC m=+586.266743692" watchObservedRunningTime="2026-07-10 16:04:57.199006916 +0000 UTC m=+586.269824988" Jul 10 16:05:02.163308 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:02.163278 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a6c8ffd-3477-4bd1-8416-a12ff141e3af" containerID="e171a24e0149b9fe903cedf6b3999c7e4c3acea64ad719400e1a2b5d547ad35f" exitCode=0 Jul 10 16:05:02.163686 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:02.163346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" event={"ID":"7a6c8ffd-3477-4bd1-8416-a12ff141e3af","Type":"ContainerDied","Data":"e171a24e0149b9fe903cedf6b3999c7e4c3acea64ad719400e1a2b5d547ad35f"} Jul 10 16:05:02.163742 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:02.163728 2576 scope.go:117] "RemoveContainer" containerID="e171a24e0149b9fe903cedf6b3999c7e4c3acea64ad719400e1a2b5d547ad35f" Jul 10 16:05:02.164631 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:02.164604 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:05:02.164695 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:02.164682 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:05:02.164739 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:02.164728 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:05:03.148002 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:03.147972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c4df6cbf-bdw4k" Jul 10 16:05:03.171120 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:03.171086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" event={"ID":"7a6c8ffd-3477-4bd1-8416-a12ff141e3af","Type":"ContainerStarted","Data":"a42dc806f69c0fecfd97385b6bcfd3e9d7f5f916509744e093a90a81c39eb77c"} Jul 10 16:05:03.430104 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:03.430018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:05:03.430284 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:03.430213 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q7p9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 16:05:03.430348 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:03.430295 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" podUID="7a6c8ffd-3477-4bd1-8416-a12ff141e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 16:05:04.163161 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:04.163126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:04.316869 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:04.316833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:04.429298 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:04.429190 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q7p9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 16:05:04.429298 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:04.429281 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" podUID="7a6c8ffd-3477-4bd1-8416-a12ff141e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 16:05:05.429888 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:05.429855 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q7p9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 16:05:05.430316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:05.429911 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" podUID="7a6c8ffd-3477-4bd1-8416-a12ff141e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 16:05:06.429384 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:06.429345 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q7p9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 16:05:06.429564 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:06.429405 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" podUID="7a6c8ffd-3477-4bd1-8416-a12ff141e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 16:05:07.433749 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:07.433717 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:05:07.434189 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:07.434072 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:05:07.434765 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:07.434748 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q7p9j" Jul 10 16:05:08.144391 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:08.144359 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-fq6hb" Jul 10 16:05:08.144559 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:08.144461 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-9v2vw" Jul 10 16:05:11.360020 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:11.359989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:05:11.360491 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:11.360298 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:05:11.363675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:11.363653 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:05:11.363814 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:11.363790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:05:13.744317 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.744282 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd"] Jul 10 16:05:13.749026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.748996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.751136 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.751111 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8j9dd\"" Jul 10 16:05:13.751372 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.751226 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 16:05:13.751947 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.751930 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 16:05:13.752036 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.751931 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 16:05:13.760967 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.760946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd"] Jul 10 16:05:13.775753 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.775720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:13.855518 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.855487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwtm\" (UniqueName: \"kubernetes.io/projected/279230e9-3502-4c02-ac04-52fe7c0441cb-kube-api-access-7pwtm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.855672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.855548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.855672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.855584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279230e9-3502-4c02-ac04-52fe7c0441cb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.855672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.855602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.855672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.855640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.855672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.855665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956345 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956528 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956528 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwtm\" (UniqueName: \"kubernetes.io/projected/279230e9-3502-4c02-ac04-52fe7c0441cb-kube-api-access-7pwtm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956602 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956602 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279230e9-3502-4c02-ac04-52fe7c0441cb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956696 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956696 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956939 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.956939 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.956932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.958765 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.958744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279230e9-3502-4c02-ac04-52fe7c0441cb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.959094 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.959077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279230e9-3502-4c02-ac04-52fe7c0441cb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:13.965531 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:13.965508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwtm\" (UniqueName: \"kubernetes.io/projected/279230e9-3502-4c02-ac04-52fe7c0441cb-kube-api-access-7pwtm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd\" (UID: \"279230e9-3502-4c02-ac04-52fe7c0441cb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:14.060929 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:14.060846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:14.204897 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:14.204868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd"] Jul 10 16:05:14.206935 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:05:14.206908 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279230e9_3502_4c02_ac04_52fe7c0441cb.slice/crio-73c2131e542282a541ab244b98781d7d4fe83c55f4dd055ea755ab167c39190e WatchSource:0}: Error finding container 73c2131e542282a541ab244b98781d7d4fe83c55f4dd055ea755ab167c39190e: Status 404 returned error can't find the container with id 73c2131e542282a541ab244b98781d7d4fe83c55f4dd055ea755ab167c39190e Jul 10 16:05:14.235238 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:14.235207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" event={"ID":"279230e9-3502-4c02-ac04-52fe7c0441cb","Type":"ContainerStarted","Data":"73c2131e542282a541ab244b98781d7d4fe83c55f4dd055ea755ab167c39190e"} Jul 10 16:05:16.247380 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:16.247345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" event={"ID":"279230e9-3502-4c02-ac04-52fe7c0441cb","Type":"ContainerStarted","Data":"d93df673c34cd0827452d36676f2bb1a053ea7bdaadb670398381ce8f0eaf637"} Jul 10 16:05:16.271178 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:16.271124 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" podStartSLOduration=1.931817407 podStartE2EDuration="3.27110709s" totalImagesPullingTime="1.339289683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:05:16.269512853 +0000 UTC m=+605.340330926" watchObservedRunningTime="2026-07-10 16:05:16.27110709 +0000 UTC m=+605.341925160" Jul 10 16:05:19.967950 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:19.967914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:20.039212 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:20.039174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:22.967313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:22.967269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:24.061738 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:24.061694 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:24.062186 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:24.061764 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:24.074415 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:24.074389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:24.299315 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:24.299283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd" Jul 10 16:05:32.136631 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.136556 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk"] Jul 10 16:05:32.143106 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.143084 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.145210 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.145185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 16:05:32.151561 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.151535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk"] Jul 10 16:05:32.165079 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.165047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:05:32.230181 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.230149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.230373 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.230200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.230373 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.230303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.230373 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.230336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkfhx\" (UniqueName: \"kubernetes.io/projected/97d58833-9980-48cc-887b-766a0b86044d-kube-api-access-gkfhx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.230373 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.230357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.230570 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.230379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d58833-9980-48cc-887b-766a0b86044d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.330823 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.330791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.330835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfhx\" (UniqueName: \"kubernetes.io/projected/97d58833-9980-48cc-887b-766a0b86044d-kube-api-access-gkfhx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.330855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.330924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d58833-9980-48cc-887b-766a0b86044d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.330984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331226 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.331039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331343 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.331323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331500 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.331477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.331583 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.331551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.333359 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.333340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d58833-9980-48cc-887b-766a0b86044d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.333530 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.333514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d58833-9980-48cc-887b-766a0b86044d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.337709 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.337689 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfhx\" (UniqueName: \"kubernetes.io/projected/97d58833-9980-48cc-887b-766a0b86044d-kube-api-access-gkfhx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhsmk\" (UID: \"97d58833-9980-48cc-887b-766a0b86044d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.455784 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.455743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:32.593461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.593432 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk"] Jul 10 16:05:32.595766 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:05:32.595739 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d58833_9980_48cc_887b_766a0b86044d.slice/crio-83d2f47e5adff01c0d2af7851c762e67e4aae3268ef67f769b000a5f2d2abc45 WatchSource:0}: Error finding container 83d2f47e5adff01c0d2af7851c762e67e4aae3268ef67f769b000a5f2d2abc45: Status 404 returned error can't find the container with id 83d2f47e5adff01c0d2af7851c762e67e4aae3268ef67f769b000a5f2d2abc45 Jul 10 16:05:32.597207 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:32.597188 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:05:33.330884 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:33.330835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" event={"ID":"97d58833-9980-48cc-887b-766a0b86044d","Type":"ContainerStarted","Data":"b5aa39c82574fbd3232b8cbf1ef2d63af07cb49691cc4f1c841c2ceb886f046f"} Jul 10 16:05:33.331404 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:33.330937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" event={"ID":"97d58833-9980-48cc-887b-766a0b86044d","Type":"ContainerStarted","Data":"83d2f47e5adff01c0d2af7851c762e67e4aae3268ef67f769b000a5f2d2abc45"} Jul 10 16:05:33.354153 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:33.354098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" podStartSLOduration=1.097932229 podStartE2EDuration="1.354080714s" totalImagesPullingTime="256.148485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:05:33.35238077 +0000 UTC m=+622.423198841" watchObservedRunningTime="2026-07-10 16:05:33.354080714 +0000 UTC m=+622.424898784" Jul 10 16:05:42.456452 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:42.456413 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:42.456849 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:42.456487 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:42.468828 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:42.468804 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:05:43.396167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:05:43.396135 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhsmk" Jul 10 16:06:00.983321 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:00.983287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:06:11.621881 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:11.621846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:06:19.865478 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.865448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz"] Jul 10 16:06:19.869409 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.869389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:19.871461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.871439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 16:06:19.882644 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.882623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz"] Jul 10 16:06:19.895731 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.895692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:06:19.960522 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.960489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:19.960695 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.960567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:19.960695 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.960591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5552p\" (UniqueName: \"kubernetes.io/projected/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-kube-api-access-5552p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:19.960695 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.960612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:19.960695 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.960637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:19.960838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:19.960746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.061894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.061854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062090 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.061924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062090 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.061984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062090 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.062056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062090 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.062078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5552p\" (UniqueName: \"kubernetes.io/projected/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-kube-api-access-5552p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062393 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.062106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062393 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.062345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062504 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.062386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.062504 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.062429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.064437 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.064417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.064788 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.064770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.070348 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.070327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5552p\" (UniqueName: \"kubernetes.io/projected/bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e-kube-api-access-5552p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6bbrz\" (UID: \"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.180416 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.180316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:20.326704 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.326673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz"] Jul 10 16:06:20.330226 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:06:20.330190 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcb0dec_a8e4_43b7_ad9b_06cdd697ea9e.slice/crio-4a2f7b659f836a22fa79be8893ea39229888e838b5c2eb81c405e0a420f4a7e6 WatchSource:0}: Error finding container 4a2f7b659f836a22fa79be8893ea39229888e838b5c2eb81c405e0a420f4a7e6: Status 404 returned error can't find the container with id 4a2f7b659f836a22fa79be8893ea39229888e838b5c2eb81c405e0a420f4a7e6 Jul 10 16:06:20.567167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:20.567143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" event={"ID":"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e","Type":"ContainerStarted","Data":"4a2f7b659f836a22fa79be8893ea39229888e838b5c2eb81c405e0a420f4a7e6"} Jul 10 16:06:21.572900 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:21.572861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" event={"ID":"bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e","Type":"ContainerStarted","Data":"2e2a2be16748c24f1e65493893d12f7c8d90eeb9e9fbb099d789729273534f22"} Jul 10 16:06:21.596533 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:21.596478 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" podStartSLOduration=2.385646264 podStartE2EDuration="2.596462019s" totalImagesPullingTime="210.815755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:06:21.594667144 +0000 UTC m=+670.665485412" watchObservedRunningTime="2026-07-10 16:06:21.596462019 +0000 UTC m=+670.667280090" Jul 10 16:06:30.180635 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:30.180597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:30.181026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:30.180661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:30.193537 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:30.193509 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:30.629893 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:30.629868 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6bbrz" Jul 10 16:06:34.737695 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:34.737656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:06:50.145002 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:50.144905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:06:50.246069 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:06:50.246035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:25.908949 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:25.908906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:33.894645 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:33.894558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:34.006470 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:34.006420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:35.155113 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:35.155082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:35.361000 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:35.360964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:38.973776 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:38.973734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:08:39.056637 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:08:39.056598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:09:47.770021 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:47.769977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:09:47.892319 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:47.892286 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787666694b-hnvgr"] Jul 10 16:09:47.892491 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:47.892466 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-787666694b-hnvgr" podUID="9249bc9e-2486-4e3f-ad63-9312dbd4d13c" containerName="manager" containerID="cri-o://e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf" gracePeriod=30 Jul 10 16:09:48.143072 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.143043 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:09:48.179326 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.179297 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-webhook-cert\") pod \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " Jul 10 16:09:48.179495 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.179356 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngt4g\" (UniqueName: \"kubernetes.io/projected/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-kube-api-access-ngt4g\") pod \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\" (UID: \"9249bc9e-2486-4e3f-ad63-9312dbd4d13c\") " Jul 10 16:09:48.182106 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.182068 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-kube-api-access-ngt4g" (OuterVolumeSpecName: "kube-api-access-ngt4g") pod "9249bc9e-2486-4e3f-ad63-9312dbd4d13c" (UID: "9249bc9e-2486-4e3f-ad63-9312dbd4d13c"). InnerVolumeSpecName "kube-api-access-ngt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:09:48.182106 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.182079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9249bc9e-2486-4e3f-ad63-9312dbd4d13c" (UID: "9249bc9e-2486-4e3f-ad63-9312dbd4d13c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:09:48.280843 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.280804 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-webhook-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:09:48.280843 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.280841 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngt4g\" (UniqueName: \"kubernetes.io/projected/9249bc9e-2486-4e3f-ad63-9312dbd4d13c-kube-api-access-ngt4g\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:09:48.594332 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.594296 2576 generic.go:356] "Generic (PLEG): container finished" podID="9249bc9e-2486-4e3f-ad63-9312dbd4d13c" containerID="e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf" exitCode=0 Jul 10 16:09:48.594521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.594342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787666694b-hnvgr" event={"ID":"9249bc9e-2486-4e3f-ad63-9312dbd4d13c","Type":"ContainerDied","Data":"e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf"} Jul 10 16:09:48.594521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.594354 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787666694b-hnvgr" Jul 10 16:09:48.594521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.594387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787666694b-hnvgr" event={"ID":"9249bc9e-2486-4e3f-ad63-9312dbd4d13c","Type":"ContainerDied","Data":"bbc429c1dc589be89c08f970b8de44ff20b05ff37e706d2756d0e5fd0840cd00"} Jul 10 16:09:48.594521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.594409 2576 scope.go:117] "RemoveContainer" containerID="e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf" Jul 10 16:09:48.604192 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.604175 2576 scope.go:117] "RemoveContainer" containerID="e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf" Jul 10 16:09:48.604485 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:09:48.604462 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf\": container with ID starting with e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf not found: ID does not exist" containerID="e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf" Jul 10 16:09:48.604547 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.604494 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf"} err="failed to get container status \"e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf\": rpc error: code = NotFound desc = could not find container \"e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf\": container with ID starting with e3a8afd8ec8c3fdf344d17088320926c4de33e56dddfd22b2db2df7567eb35cf not found: ID does not exist" Jul 10 16:09:48.627591 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.627564 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787666694b-hnvgr"] Jul 10 16:09:48.634712 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:48.634690 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-787666694b-hnvgr"] Jul 10 16:09:49.364596 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.364557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:09:49.495617 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.495584 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9249bc9e-2486-4e3f-ad63-9312dbd4d13c" path="/var/lib/kubelet/pods/9249bc9e-2486-4e3f-ad63-9312dbd4d13c/volumes" Jul 10 16:09:49.498519 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.498492 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787666694b-8jkpg"] Jul 10 16:09:49.498936 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.498919 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9249bc9e-2486-4e3f-ad63-9312dbd4d13c" containerName="manager" Jul 10 16:09:49.499012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.498937 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9249bc9e-2486-4e3f-ad63-9312dbd4d13c" containerName="manager" Jul 10 16:09:49.499062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.499042 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9249bc9e-2486-4e3f-ad63-9312dbd4d13c" containerName="manager" Jul 10 16:09:49.503233 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.503212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.506862 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.506834 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 16:09:49.506993 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.506835 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d77qb\"" Jul 10 16:09:49.506993 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.506880 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 16:09:49.517091 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.517068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787666694b-8jkpg"] Jul 10 16:09:49.592379 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.592342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktfb\" (UniqueName: \"kubernetes.io/projected/42ac1655-2931-4170-a3d8-ccc9f1bd198c-kube-api-access-4ktfb\") pod \"maas-controller-787666694b-8jkpg\" (UID: \"42ac1655-2931-4170-a3d8-ccc9f1bd198c\") " pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.592561 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.592494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ac1655-2931-4170-a3d8-ccc9f1bd198c-webhook-cert\") pod \"maas-controller-787666694b-8jkpg\" (UID: \"42ac1655-2931-4170-a3d8-ccc9f1bd198c\") " pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.693612 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.693579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktfb\" (UniqueName: \"kubernetes.io/projected/42ac1655-2931-4170-a3d8-ccc9f1bd198c-kube-api-access-4ktfb\") pod \"maas-controller-787666694b-8jkpg\" (UID: \"42ac1655-2931-4170-a3d8-ccc9f1bd198c\") " pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.693798 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.693697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ac1655-2931-4170-a3d8-ccc9f1bd198c-webhook-cert\") pod \"maas-controller-787666694b-8jkpg\" (UID: \"42ac1655-2931-4170-a3d8-ccc9f1bd198c\") " pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.696312 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.696289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ac1655-2931-4170-a3d8-ccc9f1bd198c-webhook-cert\") pod \"maas-controller-787666694b-8jkpg\" (UID: \"42ac1655-2931-4170-a3d8-ccc9f1bd198c\") " pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.707787 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.707762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktfb\" (UniqueName: \"kubernetes.io/projected/42ac1655-2931-4170-a3d8-ccc9f1bd198c-kube-api-access-4ktfb\") pod \"maas-controller-787666694b-8jkpg\" (UID: \"42ac1655-2931-4170-a3d8-ccc9f1bd198c\") " pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.814002 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.813965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:49.969208 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:49.969178 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787666694b-8jkpg"] Jul 10 16:09:49.970490 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:09:49.970461 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ac1655_2931_4170_a3d8_ccc9f1bd198c.slice/crio-2756570ea86f938fc518a4a04b7fb0abe77858b547092b7b690cfa83091f4e6a WatchSource:0}: Error finding container 2756570ea86f938fc518a4a04b7fb0abe77858b547092b7b690cfa83091f4e6a: Status 404 returned error can't find the container with id 2756570ea86f938fc518a4a04b7fb0abe77858b547092b7b690cfa83091f4e6a Jul 10 16:09:50.607243 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:50.607211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787666694b-8jkpg" event={"ID":"42ac1655-2931-4170-a3d8-ccc9f1bd198c","Type":"ContainerStarted","Data":"fe5ccf8a430186ecbb7962086d4dfd1c227a05e4f98c52272abe5f0dbeadad8d"} Jul 10 16:09:50.607642 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:50.607293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787666694b-8jkpg" event={"ID":"42ac1655-2931-4170-a3d8-ccc9f1bd198c","Type":"ContainerStarted","Data":"2756570ea86f938fc518a4a04b7fb0abe77858b547092b7b690cfa83091f4e6a"} Jul 10 16:09:50.607642 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:50.607410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:09:50.638196 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:09:50.638148 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787666694b-8jkpg" podStartSLOduration=1.121982074 podStartE2EDuration="1.638131899s" totalImagesPullingTime="516.149825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:09:50.63500168 +0000 UTC m=+879.705819763" watchObservedRunningTime="2026-07-10 16:09:50.638131899 +0000 UTC m=+879.708950000" Jul 10 16:10:01.619958 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:01.619875 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787666694b-8jkpg" Jul 10 16:10:11.390407 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:11.390375 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:10:11.394197 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:11.394172 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:10:11.395134 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:11.395115 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:10:11.398751 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:11.398721 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:10:26.357922 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:26.357876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:26.440164 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:26.440131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:26.593972 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:26.593925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:32.832623 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:32.832590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:32.946685 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:32.946651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:33.043505 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:33.043467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:33.217521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:33.217489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:39.113420 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:39.113379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:10:40.541902 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:10:40.541852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:11:56.185927 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:11:56.185818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:14:43.700305 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:43.700203 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh"] Jul 10 16:14:43.701027 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:43.700482 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" podUID="43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" containerName="manager" containerID="cri-o://d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429" gracePeriod=10 Jul 10 16:14:46.358478 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.358448 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:14:46.447770 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.447734 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c96r4\" (UniqueName: \"kubernetes.io/projected/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-kube-api-access-c96r4\") pod \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " Jul 10 16:14:46.447965 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.447800 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-extensions-socket-volume\") pod \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\" (UID: \"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760\") " Jul 10 16:14:46.448204 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.448177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" (UID: "43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:14:46.449941 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.449920 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-kube-api-access-c96r4" (OuterVolumeSpecName: "kube-api-access-c96r4") pod "43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" (UID: "43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760"). InnerVolumeSpecName "kube-api-access-c96r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:14:46.548757 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.548721 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96r4\" (UniqueName: \"kubernetes.io/projected/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-kube-api-access-c96r4\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:14:46.548757 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:46.548753 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760-extensions-socket-volume\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:14:47.105556 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.105518 2576 generic.go:356] "Generic (PLEG): container finished" podID="43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" containerID="d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429" exitCode=0 Jul 10 16:14:47.105772 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.105566 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" Jul 10 16:14:47.105772 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.105602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" event={"ID":"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760","Type":"ContainerDied","Data":"d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429"} Jul 10 16:14:47.105772 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.105658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh" event={"ID":"43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760","Type":"ContainerDied","Data":"80f8e699d93562f56a3860a2552b9fb4505bbbc9e44fb7b2a3ce12a508b439e4"} Jul 10 16:14:47.105772 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.105680 2576 scope.go:117] "RemoveContainer" containerID="d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429" Jul 10 16:14:47.116139 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.116121 2576 scope.go:117] "RemoveContainer" containerID="d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429" Jul 10 16:14:47.116420 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:14:47.116397 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429\": container with ID starting with d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429 not found: ID does not exist" containerID="d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429" Jul 10 16:14:47.116534 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.116428 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429"} err="failed to get container status \"d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429\": rpc error: code = NotFound desc = could not find container \"d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429\": container with ID starting with d6b5f4ad8e547bb47fb1cc693e608fe7e5a74d80841483d9575bae1f95310429 not found: ID does not exist" Jul 10 16:14:47.127450 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.127420 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh"] Jul 10 16:14:47.134685 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.134663 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p4sdh"] Jul 10 16:14:47.495712 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:14:47.495682 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" path="/var/lib/kubelet/pods/43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760/volumes" Jul 10 16:15:00.151139 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.151105 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn"] Jul 10 16:15:00.151580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.151520 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" containerName="manager" Jul 10 16:15:00.151580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.151532 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" containerName="manager" Jul 10 16:15:00.151653 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.151609 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d764f0-f4a3-43dc-bf8b-fbd1a0f4c760" containerName="manager" Jul 10 16:15:00.156434 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.156416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" Jul 10 16:15:00.158858 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.158837 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rf26q\"" Jul 10 16:15:00.161721 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.161701 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn"] Jul 10 16:15:00.168074 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.168055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" Jul 10 16:15:00.309120 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.309091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn"] Jul 10 16:15:00.311938 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:15:00.311913 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9899c1d6_5ce2_4230_81b0_648b0bfc5f3b.slice/crio-307635974e56dd4aa202013f2a998377df392da7a7604f7388389a867774f81c WatchSource:0}: Error finding container 307635974e56dd4aa202013f2a998377df392da7a7604f7388389a867774f81c: Status 404 returned error can't find the container with id 307635974e56dd4aa202013f2a998377df392da7a7604f7388389a867774f81c Jul 10 16:15:00.314301 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:00.314285 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:15:01.180296 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:01.180238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" event={"ID":"9899c1d6-5ce2-4230-81b0-648b0bfc5f3b","Type":"ContainerStarted","Data":"307635974e56dd4aa202013f2a998377df392da7a7604f7388389a867774f81c"} Jul 10 16:15:03.192818 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:03.192779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" event={"ID":"9899c1d6-5ce2-4230-81b0-648b0bfc5f3b","Type":"ContainerStarted","Data":"014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a"} Jul 10 16:15:03.209245 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:03.209192 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" podStartSLOduration=1.145794388 podStartE2EDuration="3.209175444s" totalImagesPullingTime="2.063381056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:15:03.20700871 +0000 UTC m=+1192.277826780" watchObservedRunningTime="2026-07-10 16:15:03.209175444 +0000 UTC m=+1192.279993514" Jul 10 16:15:11.422581 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:11.422464 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:15:11.426232 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:11.426208 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:15:11.427204 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:11.427185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:15:11.430547 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:11.430530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:15:22.172511 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:22.172476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:15:44.900943 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:44.900904 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b"] Jul 10 16:15:44.904705 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:44.904684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:44.906938 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:44.906917 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h468v\"" Jul 10 16:15:44.915729 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:44.915707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b"] Jul 10 16:15:44.969574 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:44.969545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062169f4-e0bc-4584-8aa9-26253f1ef984-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fs64b\" (UID: \"062169f4-e0bc-4584-8aa9-26253f1ef984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:44.969574 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:44.969577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrlp\" (UniqueName: \"kubernetes.io/projected/062169f4-e0bc-4584-8aa9-26253f1ef984-kube-api-access-gkrlp\") pod \"kuadrant-operator-controller-manager-86878c769f-fs64b\" (UID: \"062169f4-e0bc-4584-8aa9-26253f1ef984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:45.070585 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.070550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062169f4-e0bc-4584-8aa9-26253f1ef984-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fs64b\" (UID: \"062169f4-e0bc-4584-8aa9-26253f1ef984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:45.070585 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.070587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrlp\" (UniqueName: \"kubernetes.io/projected/062169f4-e0bc-4584-8aa9-26253f1ef984-kube-api-access-gkrlp\") pod \"kuadrant-operator-controller-manager-86878c769f-fs64b\" (UID: \"062169f4-e0bc-4584-8aa9-26253f1ef984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:45.070939 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.070920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062169f4-e0bc-4584-8aa9-26253f1ef984-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fs64b\" (UID: \"062169f4-e0bc-4584-8aa9-26253f1ef984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:45.078241 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.078210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrlp\" (UniqueName: \"kubernetes.io/projected/062169f4-e0bc-4584-8aa9-26253f1ef984-kube-api-access-gkrlp\") pod \"kuadrant-operator-controller-manager-86878c769f-fs64b\" (UID: \"062169f4-e0bc-4584-8aa9-26253f1ef984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:45.215167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.215131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:45.356507 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.356472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b"] Jul 10 16:15:45.360565 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:15:45.360539 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062169f4_e0bc_4584_8aa9_26253f1ef984.slice/crio-1d2f2973bb1e34ef1e443605caeaedf33a0a037ad16c301954c80c31c2152728 WatchSource:0}: Error finding container 1d2f2973bb1e34ef1e443605caeaedf33a0a037ad16c301954c80c31c2152728: Status 404 returned error can't find the container with id 1d2f2973bb1e34ef1e443605caeaedf33a0a037ad16c301954c80c31c2152728 Jul 10 16:15:45.403951 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:45.403916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" event={"ID":"062169f4-e0bc-4584-8aa9-26253f1ef984","Type":"ContainerStarted","Data":"1d2f2973bb1e34ef1e443605caeaedf33a0a037ad16c301954c80c31c2152728"} Jul 10 16:15:46.410881 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:46.410837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" event={"ID":"062169f4-e0bc-4584-8aa9-26253f1ef984","Type":"ContainerStarted","Data":"edb108ed61a5736099e6af908f8399d210839480783dea2ae07b7c32b93ee988"} Jul 10 16:15:46.411326 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:46.410971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:15:46.430818 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:46.430761 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" podStartSLOduration=2.430746185 podStartE2EDuration="2.430746185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:15:46.429482219 +0000 UTC m=+1235.500300290" watchObservedRunningTime="2026-07-10 16:15:46.430746185 +0000 UTC m=+1235.501564253" Jul 10 16:15:57.419390 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:15:57.419352 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fs64b" Jul 10 16:16:00.011289 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:00.011238 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn"] Jul 10 16:16:00.011736 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:00.011471 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" podUID="9899c1d6-5ce2-4230-81b0-648b0bfc5f3b" containerName="cleanup" containerID="cri-o://014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a" gracePeriod=30 Jul 10 16:16:30.210033 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.210000 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" Jul 10 16:16:30.635415 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.635381 2576 generic.go:356] "Generic (PLEG): container finished" podID="9899c1d6-5ce2-4230-81b0-648b0bfc5f3b" containerID="014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a" exitCode=137 Jul 10 16:16:30.635593 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.635426 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" Jul 10 16:16:30.635593 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.635469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" event={"ID":"9899c1d6-5ce2-4230-81b0-648b0bfc5f3b","Type":"ContainerDied","Data":"014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a"} Jul 10 16:16:30.635593 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.635527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn" event={"ID":"9899c1d6-5ce2-4230-81b0-648b0bfc5f3b","Type":"ContainerDied","Data":"307635974e56dd4aa202013f2a998377df392da7a7604f7388389a867774f81c"} Jul 10 16:16:30.635593 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.635555 2576 scope.go:117] "RemoveContainer" containerID="014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a" Jul 10 16:16:30.644281 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.644241 2576 scope.go:117] "RemoveContainer" containerID="014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a" Jul 10 16:16:30.644526 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:16:30.644511 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a\": container with ID starting with 014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a not found: ID does not exist" containerID="014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a" Jul 10 16:16:30.644582 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.644536 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a"} err="failed to get container status \"014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a\": rpc error: code = NotFound desc = could not find container \"014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a\": container with ID starting with 014fff1f49e7ab9e5decf30bd0b9a2e0d4219fc48da7c035a83e47c9bde28f6a not found: ID does not exist" Jul 10 16:16:30.659682 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.659647 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn"] Jul 10 16:16:30.663362 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:30.663339 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728335-rkntn"] Jul 10 16:16:31.495580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:16:31.495539 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9899c1d6-5ce2-4230-81b0-648b0bfc5f3b" path="/var/lib/kubelet/pods/9899c1d6-5ce2-4230-81b0-648b0bfc5f3b/volumes" Jul 10 16:17:49.835454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:17:49.835377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:17:51.768130 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:17:51.768073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:17:55.630319 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:17:55.630280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:19:53.388588 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:19:53.388544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:20:04.130470 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:20:04.130439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:20:08.067959 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:20:08.067923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:20:11.452443 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:20:11.452411 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:20:11.456235 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:20:11.456213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:20:11.458616 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:20:11.458589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:20:11.461950 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:20:11.461933 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:21:39.015687 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:21:39.015645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:21:45.884531 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:21:45.884486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:21:54.172654 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:21:54.172610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:02.444511 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:02.444419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:07.574744 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:07.574705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:10.480388 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:10.480352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:10.580859 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:10.580816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:10.928524 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:10.928438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:16.849924 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:16.849881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:16.950575 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:16.950536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:17.030657 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.030623 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h"] Jul 10 16:22:17.031116 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.031097 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9899c1d6-5ce2-4230-81b0-648b0bfc5f3b" containerName="cleanup" Jul 10 16:22:17.031116 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.031115 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9899c1d6-5ce2-4230-81b0-648b0bfc5f3b" containerName="cleanup" Jul 10 16:22:17.031222 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.031211 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9899c1d6-5ce2-4230-81b0-648b0bfc5f3b" containerName="cleanup" Jul 10 16:22:17.034338 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.034307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.036416 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.036393 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b056e8e8-openshift-default-dockercfg-zv6fh\"" Jul 10 16:22:17.045426 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-data\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045554 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-podinfo\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045554 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-envoy\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045554 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istiod-ca-cert\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045554 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-socket\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045776 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-credential-socket\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045776 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-token\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045776 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-certs\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.045776 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.045749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqsq\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-kube-api-access-7wqsq\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.059313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.059284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h"] Jul 10 16:22:17.073937 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.073875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:17.146580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-token\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-certs\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqsq\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-kube-api-access-7wqsq\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-data\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-podinfo\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-envoy\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istiod-ca-cert\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-socket\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.146886 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.146781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-credential-socket\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.147641 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.147616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-certs\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.147815 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.147777 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-credential-socket\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.147948 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.147659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-data\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.148012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.147938 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-socket\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.148070 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.148057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istiod-ca-cert\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.149925 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.149898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-envoy\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.150317 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.150298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-podinfo\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.153268 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.153235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:17.154633 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.154614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-token\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.154832 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.154814 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqsq\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-kube-api-access-7wqsq\") pod \"e2e-ait-b056e8e8-openshift-default-86f8448545-q959h\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.347982 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.347933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:17.494452 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:22:17.494418 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd6ed7b_fa82_4522_b4ba_1f9312fe0c09.slice/crio-eb82c2772187adf1a2024d1dd5fd3c1cd30f56025d66acb7be274fe15b2f6dd5 WatchSource:0}: Error finding container eb82c2772187adf1a2024d1dd5fd3c1cd30f56025d66acb7be274fe15b2f6dd5: Status 404 returned error can't find the container with id eb82c2772187adf1a2024d1dd5fd3c1cd30f56025d66acb7be274fe15b2f6dd5 Jul 10 16:22:17.496673 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.496640 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:22:17.496782 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.496718 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:22:17.496782 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.496758 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:22:17.497398 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:17.497348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h"] Jul 10 16:22:18.258990 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.258942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:18.335104 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.335070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:18.335784 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.335756 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5"] Jul 10 16:22:18.339404 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.339383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.341453 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.341432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b056e8e8-serving-cert\"" Jul 10 16:22:18.349224 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.349196 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5"] Jul 10 16:22:18.363066 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.363039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chk6m\" (UniqueName: \"kubernetes.io/projected/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-kube-api-access-chk6m\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.363184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.363072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.363184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.363102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-openshift-service-ca\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.393362 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.393331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:18.398076 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.398046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" event={"ID":"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09","Type":"ContainerStarted","Data":"a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495"} Jul 10 16:22:18.398197 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.398108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" event={"ID":"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09","Type":"ContainerStarted","Data":"eb82c2772187adf1a2024d1dd5fd3c1cd30f56025d66acb7be274fe15b2f6dd5"} Jul 10 16:22:18.416429 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.416382 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" podStartSLOduration=1.416365138 podStartE2EDuration="1.416365138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:22:18.415198236 +0000 UTC m=+1627.486016305" watchObservedRunningTime="2026-07-10 16:22:18.416365138 +0000 UTC m=+1627.487183208" Jul 10 16:22:18.464379 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.463971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chk6m\" (UniqueName: \"kubernetes.io/projected/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-kube-api-access-chk6m\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.464379 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.464038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.464379 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.464128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-openshift-service-ca\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.464379 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:18.464286 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-serving-cert: secret "maas-api-e2e-ait-b056e8e8-serving-cert" not found Jul 10 16:22:18.464379 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:18.464374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls podName:5eb0f7ca-03d2-4db3-a11d-1818a1551a37 nodeName:}" failed. No retries permitted until 2026-07-10 16:22:18.964350634 +0000 UTC m=+1628.035168682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls") pod "maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" (UID: "5eb0f7ca-03d2-4db3-a11d-1818a1551a37") : secret "maas-api-e2e-ait-b056e8e8-serving-cert" not found Jul 10 16:22:18.464979 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.464956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-openshift-service-ca\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.473483 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.473455 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chk6m\" (UniqueName: \"kubernetes.io/projected/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-kube-api-access-chk6m\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.539449 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.539351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:18.801296 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.801180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:18.900167 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.900135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:18.969841 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.969799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:18.972512 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:18.972479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls\") pod \"maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:19.251607 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.251569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:19.348643 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.348617 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:19.354074 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.354049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:19.400075 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.400039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5"] Jul 10 16:22:19.408118 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:22:19.408090 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb0f7ca_03d2_4db3_a11d_1818a1551a37.slice/crio-816e506ee412cd47f1b336da4096edfe6786217be522bac568287fa1bfbc4d23 WatchSource:0}: Error finding container 816e506ee412cd47f1b336da4096edfe6786217be522bac568287fa1bfbc4d23: Status 404 returned error can't find the container with id 816e506ee412cd47f1b336da4096edfe6786217be522bac568287fa1bfbc4d23 Jul 10 16:22:19.408316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.408281 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:19.409475 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.409457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:19.409599 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:19.409583 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:22:20.414838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:20.414736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" event={"ID":"5eb0f7ca-03d2-4db3-a11d-1818a1551a37","Type":"ContainerStarted","Data":"3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd"} Jul 10 16:22:20.414838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:20.414796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" event={"ID":"5eb0f7ca-03d2-4db3-a11d-1818a1551a37","Type":"ContainerStarted","Data":"816e506ee412cd47f1b336da4096edfe6786217be522bac568287fa1bfbc4d23"} Jul 10 16:22:20.415294 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:20.414941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:20.433507 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:20.433459 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" podStartSLOduration=1.8546342230000001 podStartE2EDuration="2.433445238s" totalImagesPullingTime="578.811015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:22:20.43143246 +0000 UTC m=+1629.502250551" watchObservedRunningTime="2026-07-10 16:22:20.433445238 +0000 UTC m=+1629.504263311" Jul 10 16:22:21.152796 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:21.152759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:23.293478 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.293422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:23.470825 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.470788 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h"] Jul 10 16:22:23.471044 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.471019 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" podUID="fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" containerName="istio-proxy" containerID="cri-o://a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495" gracePeriod=30 Jul 10 16:22:23.509157 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.509106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:23.590873 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.590729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:23.761118 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.761088 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5"] Jul 10 16:22:23.761445 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.761409 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" podUID="5eb0f7ca-03d2-4db3-a11d-1818a1551a37" containerName="maas-api" containerID="cri-o://3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd" gracePeriod=30 Jul 10 16:22:23.788608 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:23.788567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:24.056652 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.056627 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:24.119178 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.119143 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls\") pod \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " Jul 10 16:22:24.119365 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.119196 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chk6m\" (UniqueName: \"kubernetes.io/projected/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-kube-api-access-chk6m\") pod \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " Jul 10 16:22:24.119365 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.119232 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-openshift-service-ca\") pod \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\" (UID: \"5eb0f7ca-03d2-4db3-a11d-1818a1551a37\") " Jul 10 16:22:24.119617 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.119590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5eb0f7ca-03d2-4db3-a11d-1818a1551a37" (UID: "5eb0f7ca-03d2-4db3-a11d-1818a1551a37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:22:24.121607 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.121579 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5eb0f7ca-03d2-4db3-a11d-1818a1551a37" (UID: "5eb0f7ca-03d2-4db3-a11d-1818a1551a37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:22:24.121730 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.121713 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-kube-api-access-chk6m" (OuterVolumeSpecName: "kube-api-access-chk6m") pod "5eb0f7ca-03d2-4db3-a11d-1818a1551a37" (UID: "5eb0f7ca-03d2-4db3-a11d-1818a1551a37"). InnerVolumeSpecName "kube-api-access-chk6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:22:24.220486 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.220450 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:24.220486 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.220481 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chk6m\" (UniqueName: \"kubernetes.io/projected/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-kube-api-access-chk6m\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:24.220486 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.220492 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb0f7ca-03d2-4db3-a11d-1818a1551a37-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:24.436934 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.436893 2576 generic.go:356] "Generic (PLEG): container finished" podID="5eb0f7ca-03d2-4db3-a11d-1818a1551a37" containerID="3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd" exitCode=0 Jul 10 16:22:24.437456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.436942 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" Jul 10 16:22:24.437456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.436984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" event={"ID":"5eb0f7ca-03d2-4db3-a11d-1818a1551a37","Type":"ContainerDied","Data":"3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd"} Jul 10 16:22:24.437456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.437043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5" event={"ID":"5eb0f7ca-03d2-4db3-a11d-1818a1551a37","Type":"ContainerDied","Data":"816e506ee412cd47f1b336da4096edfe6786217be522bac568287fa1bfbc4d23"} Jul 10 16:22:24.437456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.437066 2576 scope.go:117] "RemoveContainer" containerID="3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd" Jul 10 16:22:24.446832 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.446812 2576 scope.go:117] "RemoveContainer" containerID="3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd" Jul 10 16:22:24.447078 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:24.447063 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd\": container with ID starting with 3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd not found: ID does not exist" containerID="3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd" Jul 10 16:22:24.447125 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.447088 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd"} err="failed to get container status \"3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd\": rpc error: code = NotFound desc = could not find container \"3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd\": container with ID starting with 3dcbca588953d6e8e3251e83a5d8293b77f179193a3acc066cf007c6102e66fd not found: ID does not exist" Jul 10 16:22:24.457459 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.457429 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5"] Jul 10 16:22:24.459702 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.459680 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b056e8e8-86bd8c6cc5-nzwj5"] Jul 10 16:22:24.936129 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:24.936092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:25.499813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:25.499779 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb0f7ca-03d2-4db3-a11d-1818a1551a37" path="/var/lib/kubelet/pods/5eb0f7ca-03d2-4db3-a11d-1818a1551a37/volumes" Jul 10 16:22:29.640631 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:29.640593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:29.741505 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:29.741468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:29.867644 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:29.867611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:29.966579 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:29.966544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:30.635185 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.635159 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:30.684867 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.684837 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-credential-socket\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.684887 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-envoy\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.684932 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-data\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.684960 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-token\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.684999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-certs\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685070 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wqsq\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-kube-api-access-7wqsq\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685117 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-podinfo\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istiod-ca-cert\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685172 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-socket\") pod \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\" (UID: \"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09\") " Jul 10 16:22:30.685351 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685171 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:22:30.685821 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685520 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-credential-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.685821 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685744 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:22:30.685821 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.685785 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:22:30.686050 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.686031 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:22:30.686307 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.686277 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:22:30.688146 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.688119 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-kube-api-access-7wqsq" (OuterVolumeSpecName: "kube-api-access-7wqsq") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "kube-api-access-7wqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:22:30.688326 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.688307 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:22:30.688847 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.688828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:22:30.689159 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.689141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" (UID: "fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.786940 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istiod-ca-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.786970 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.786979 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-envoy\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.786988 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-data\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.786995 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-token\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.787002 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-workload-certs\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.787010 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqsq\" (UniqueName: \"kubernetes.io/projected/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-kube-api-access-7wqsq\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.787030 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.787020 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09-istio-podinfo\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:30.959449 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:30.959416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:31.038765 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.038666 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp"] Jul 10 16:22:31.039345 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.039324 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb0f7ca-03d2-4db3-a11d-1818a1551a37" containerName="maas-api" Jul 10 16:22:31.039345 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.039346 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb0f7ca-03d2-4db3-a11d-1818a1551a37" containerName="maas-api" Jul 10 16:22:31.039532 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.039365 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" containerName="istio-proxy" Jul 10 16:22:31.039532 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.039374 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" containerName="istio-proxy" Jul 10 16:22:31.039532 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.039480 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb0f7ca-03d2-4db3-a11d-1818a1551a37" containerName="maas-api" Jul 10 16:22:31.039532 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.039493 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" containerName="istio-proxy" Jul 10 16:22:31.042243 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.042215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.042604 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.042582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:31.044282 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.044242 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-87e54437-serving-cert\"" Jul 10 16:22:31.051126 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.051054 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp"] Jul 10 16:22:31.090321 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.090289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.090525 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.090372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b91b7b-8bea-4526-b526-0223e958c2da-openshift-service-ca\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.090525 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.090421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7jlf\" (UniqueName: \"kubernetes.io/projected/d3b91b7b-8bea-4526-b526-0223e958c2da-kube-api-access-r7jlf\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.097675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.097643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:31.191180 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.191143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.191416 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.191201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b91b7b-8bea-4526-b526-0223e958c2da-openshift-service-ca\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.191416 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.191234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7jlf\" (UniqueName: \"kubernetes.io/projected/d3b91b7b-8bea-4526-b526-0223e958c2da-kube-api-access-r7jlf\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.191416 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:31.191346 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-serving-cert: secret "maas-api-e2e-ait-87e54437-serving-cert" not found Jul 10 16:22:31.191592 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:31.191424 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls podName:d3b91b7b-8bea-4526-b526-0223e958c2da nodeName:}" failed. No retries permitted until 2026-07-10 16:22:31.691404098 +0000 UTC m=+1640.762222184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls") pod "maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" (UID: "d3b91b7b-8bea-4526-b526-0223e958c2da") : secret "maas-api-e2e-ait-87e54437-serving-cert" not found Jul 10 16:22:31.191982 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.191963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b91b7b-8bea-4526-b526-0223e958c2da-openshift-service-ca\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.204572 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.204548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7jlf\" (UniqueName: \"kubernetes.io/projected/d3b91b7b-8bea-4526-b526-0223e958c2da-kube-api-access-r7jlf\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.463896 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.463860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:31.481551 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.481520 2576 generic.go:356] "Generic (PLEG): container finished" podID="fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" containerID="a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495" exitCode=0 Jul 10 16:22:31.481726 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.481563 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" Jul 10 16:22:31.481726 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.481604 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" event={"ID":"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09","Type":"ContainerDied","Data":"a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495"} Jul 10 16:22:31.481726 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.481658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h" event={"ID":"fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09","Type":"ContainerDied","Data":"eb82c2772187adf1a2024d1dd5fd3c1cd30f56025d66acb7be274fe15b2f6dd5"} Jul 10 16:22:31.481726 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.481673 2576 scope.go:117] "RemoveContainer" containerID="a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495" Jul 10 16:22:31.492205 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.492026 2576 scope.go:117] "RemoveContainer" containerID="a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495" Jul 10 16:22:31.492360 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:31.492334 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495\": container with ID starting with a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495 not found: ID does not exist" containerID="a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495" Jul 10 16:22:31.492415 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.492370 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495"} err="failed to get container status \"a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495\": rpc error: code = NotFound desc = could not find container \"a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495\": container with ID starting with a3a787668df83153dc7cbdfc2c05320a5e65673bca73278f9fe5a9fd46de9495 not found: ID does not exist" Jul 10 16:22:31.505847 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.505816 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h"] Jul 10 16:22:31.511399 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.511374 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b056e8e8-openshift-default-86f8448545-q959h"] Jul 10 16:22:31.695699 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.695662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.698570 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.698543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls\") pod \"maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:31.714979 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.714913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:31.817130 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.817097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:31.954874 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:31.954830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:32.110683 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:32.110657 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp"] Jul 10 16:22:32.490220 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:32.490195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" event={"ID":"d3b91b7b-8bea-4526-b526-0223e958c2da","Type":"ContainerStarted","Data":"f6587fd18b1a8169485e3d8c05e9c2f8e1eacd3fa5ef56ab6a09a3f16791b696"} Jul 10 16:22:33.496643 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:33.496609 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09" path="/var/lib/kubelet/pods/fbd6ed7b-fa82-4522-b4ba-1f9312fe0c09/volumes" Jul 10 16:22:33.499313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:33.499279 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" event={"ID":"d3b91b7b-8bea-4526-b526-0223e958c2da","Type":"ContainerStarted","Data":"5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b"} Jul 10 16:22:33.499526 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:33.499401 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:33.516687 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:33.516637 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" podStartSLOduration=2.144177261 podStartE2EDuration="2.516619133s" totalImagesPullingTime="372.441872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:22:33.515317413 +0000 UTC m=+1642.586135485" watchObservedRunningTime="2026-07-10 16:22:33.516619133 +0000 UTC m=+1642.587437203" Jul 10 16:22:33.548681 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:33.548632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:36.100637 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.100595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:36.437132 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.437099 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp"] Jul 10 16:22:36.437367 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.437336 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" podUID="d3b91b7b-8bea-4526-b526-0223e958c2da" containerName="maas-api" containerID="cri-o://5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b" gracePeriod=30 Jul 10 16:22:36.484472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.484427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:36.687573 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.687495 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:36.746137 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.746102 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b91b7b-8bea-4526-b526-0223e958c2da-openshift-service-ca\") pod \"d3b91b7b-8bea-4526-b526-0223e958c2da\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " Jul 10 16:22:36.746334 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.746186 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7jlf\" (UniqueName: \"kubernetes.io/projected/d3b91b7b-8bea-4526-b526-0223e958c2da-kube-api-access-r7jlf\") pod \"d3b91b7b-8bea-4526-b526-0223e958c2da\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " Jul 10 16:22:36.746334 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.746218 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls\") pod \"d3b91b7b-8bea-4526-b526-0223e958c2da\" (UID: \"d3b91b7b-8bea-4526-b526-0223e958c2da\") " Jul 10 16:22:36.746562 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.746540 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b91b7b-8bea-4526-b526-0223e958c2da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3b91b7b-8bea-4526-b526-0223e958c2da" (UID: "d3b91b7b-8bea-4526-b526-0223e958c2da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:22:36.748472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.748447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b91b7b-8bea-4526-b526-0223e958c2da-kube-api-access-r7jlf" (OuterVolumeSpecName: "kube-api-access-r7jlf") pod "d3b91b7b-8bea-4526-b526-0223e958c2da" (UID: "d3b91b7b-8bea-4526-b526-0223e958c2da"). InnerVolumeSpecName "kube-api-access-r7jlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:22:36.748554 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.748488 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3b91b7b-8bea-4526-b526-0223e958c2da" (UID: "d3b91b7b-8bea-4526-b526-0223e958c2da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:22:36.847385 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.847344 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b91b7b-8bea-4526-b526-0223e958c2da-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:36.847385 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.847376 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7jlf\" (UniqueName: \"kubernetes.io/projected/d3b91b7b-8bea-4526-b526-0223e958c2da-kube-api-access-r7jlf\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:36.847385 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:36.847390 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b91b7b-8bea-4526-b526-0223e958c2da-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:37.518803 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.518770 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3b91b7b-8bea-4526-b526-0223e958c2da" containerID="5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b" exitCode=0 Jul 10 16:22:37.519184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.518821 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" Jul 10 16:22:37.519184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.518854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" event={"ID":"d3b91b7b-8bea-4526-b526-0223e958c2da","Type":"ContainerDied","Data":"5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b"} Jul 10 16:22:37.519184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.518906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp" event={"ID":"d3b91b7b-8bea-4526-b526-0223e958c2da","Type":"ContainerDied","Data":"f6587fd18b1a8169485e3d8c05e9c2f8e1eacd3fa5ef56ab6a09a3f16791b696"} Jul 10 16:22:37.519184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.518927 2576 scope.go:117] "RemoveContainer" containerID="5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b" Jul 10 16:22:37.527767 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.527742 2576 scope.go:117] "RemoveContainer" containerID="5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b" Jul 10 16:22:37.528014 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:37.527994 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b\": container with ID starting with 5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b not found: ID does not exist" containerID="5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b" Jul 10 16:22:37.528075 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.528021 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b"} err="failed to get container status \"5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b\": rpc error: code = NotFound desc = could not find container \"5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b\": container with ID starting with 5809cf33823783b7cc47bf66864d8ea69adcf62ab4a7ae9076402cdca345cb8b not found: ID does not exist" Jul 10 16:22:37.535627 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.535605 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp"] Jul 10 16:22:37.537363 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:37.537346 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e54437-5b9599ff4c-rphjp"] Jul 10 16:22:39.496056 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:39.496023 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b91b7b-8bea-4526-b526-0223e958c2da" path="/var/lib/kubelet/pods/d3b91b7b-8bea-4526-b526-0223e958c2da/volumes" Jul 10 16:22:42.228522 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:42.228486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:42.329325 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:42.329295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:45.229915 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:45.229876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:48.310617 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:48.310580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:48.365681 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:48.365646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:48.525533 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:48.525503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:48.625696 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:48.625608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:49.646793 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.646759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:49.760107 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.760073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:49.809226 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.809185 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt"] Jul 10 16:22:49.809704 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.809684 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b91b7b-8bea-4526-b526-0223e958c2da" containerName="maas-api" Jul 10 16:22:49.809800 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.809705 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b91b7b-8bea-4526-b526-0223e958c2da" containerName="maas-api" Jul 10 16:22:49.809800 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.809794 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b91b7b-8bea-4526-b526-0223e958c2da" containerName="maas-api" Jul 10 16:22:49.811891 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.811869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:49.815175 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.815157 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bd511652-serving-cert\"" Jul 10 16:22:49.826869 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.826847 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt"] Jul 10 16:22:49.857183 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.857155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:49.966810 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.966764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfd182-91e6-4abd-851e-a5b3462cc365-openshift-service-ca\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:49.967012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.966887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:49.967012 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:49.966926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgp8\" (UniqueName: \"kubernetes.io/projected/cbcfd182-91e6-4abd-851e-a5b3462cc365-kube-api-access-6cgp8\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.068274 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.068215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfd182-91e6-4abd-851e-a5b3462cc365-openshift-service-ca\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.068463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.068332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.068463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.068363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgp8\" (UniqueName: \"kubernetes.io/projected/cbcfd182-91e6-4abd-851e-a5b3462cc365-kube-api-access-6cgp8\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.068548 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:50.068484 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-serving-cert: secret "maas-api-e2e-derive-bd511652-serving-cert" not found Jul 10 16:22:50.068590 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:22:50.068558 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls podName:cbcfd182-91e6-4abd-851e-a5b3462cc365 nodeName:}" failed. No retries permitted until 2026-07-10 16:22:50.568539572 +0000 UTC m=+1659.639357620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls") pod "maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" (UID: "cbcfd182-91e6-4abd-851e-a5b3462cc365") : secret "maas-api-e2e-derive-bd511652-serving-cert" not found Jul 10 16:22:50.068879 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.068858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfd182-91e6-4abd-851e-a5b3462cc365-openshift-service-ca\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.098123 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.098089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgp8\" (UniqueName: \"kubernetes.io/projected/cbcfd182-91e6-4abd-851e-a5b3462cc365-kube-api-access-6cgp8\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.337443 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.337360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:50.437781 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.437747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:50.572919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.572880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.575492 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.575468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls\") pod \"maas-api-e2e-derive-bd511652-67b6fc59-4z2bt\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.722470 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.722434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:50.869007 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:50.868976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt"] Jul 10 16:22:50.871021 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:22:50.870996 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcfd182_91e6_4abd_851e_a5b3462cc365.slice/crio-d69cb95e06ebc4aecefd9fa389d3335682ce2f42c5054e52575f86ace3f8dfb4 WatchSource:0}: Error finding container d69cb95e06ebc4aecefd9fa389d3335682ce2f42c5054e52575f86ace3f8dfb4: Status 404 returned error can't find the container with id d69cb95e06ebc4aecefd9fa389d3335682ce2f42c5054e52575f86ace3f8dfb4 Jul 10 16:22:51.592285 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:51.592153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" event={"ID":"cbcfd182-91e6-4abd-851e-a5b3462cc365","Type":"ContainerStarted","Data":"05911917ffc95effe132cb5de387ce5dfd6466aa33b1cb5903b8f07b2bdb3068"} Jul 10 16:22:51.592467 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:51.592280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" event={"ID":"cbcfd182-91e6-4abd-851e-a5b3462cc365","Type":"ContainerStarted","Data":"d69cb95e06ebc4aecefd9fa389d3335682ce2f42c5054e52575f86ace3f8dfb4"} Jul 10 16:22:51.592467 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:51.592319 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:51.610087 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:51.610015 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" podStartSLOduration=2.200773942 podStartE2EDuration="2.609995575s" totalImagesPullingTime="409.221633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:22:51.609359889 +0000 UTC m=+1660.680177960" watchObservedRunningTime="2026-07-10 16:22:51.609995575 +0000 UTC m=+1660.680813647" Jul 10 16:22:51.885750 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:51.885701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:54.078981 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.078947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:54.278819 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.278792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:54.385347 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.385240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:54.441002 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.440971 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt"] Jul 10 16:22:54.441458 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.441422 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" podUID="cbcfd182-91e6-4abd-851e-a5b3462cc365" containerName="maas-api" containerID="cri-o://05911917ffc95effe132cb5de387ce5dfd6466aa33b1cb5903b8f07b2bdb3068" gracePeriod=30 Jul 10 16:22:54.469626 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.469595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:54.609949 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.609921 2576 generic.go:356] "Generic (PLEG): container finished" podID="cbcfd182-91e6-4abd-851e-a5b3462cc365" containerID="05911917ffc95effe132cb5de387ce5dfd6466aa33b1cb5903b8f07b2bdb3068" exitCode=0 Jul 10 16:22:54.610098 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.610065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" event={"ID":"cbcfd182-91e6-4abd-851e-a5b3462cc365","Type":"ContainerDied","Data":"05911917ffc95effe132cb5de387ce5dfd6466aa33b1cb5903b8f07b2bdb3068"} Jul 10 16:22:54.690362 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.690336 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:54.813678 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.813649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cgp8\" (UniqueName: \"kubernetes.io/projected/cbcfd182-91e6-4abd-851e-a5b3462cc365-kube-api-access-6cgp8\") pod \"cbcfd182-91e6-4abd-851e-a5b3462cc365\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " Jul 10 16:22:54.813890 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.813716 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfd182-91e6-4abd-851e-a5b3462cc365-openshift-service-ca\") pod \"cbcfd182-91e6-4abd-851e-a5b3462cc365\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " Jul 10 16:22:54.813890 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.813822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls\") pod \"cbcfd182-91e6-4abd-851e-a5b3462cc365\" (UID: \"cbcfd182-91e6-4abd-851e-a5b3462cc365\") " Jul 10 16:22:54.814140 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.814113 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcfd182-91e6-4abd-851e-a5b3462cc365-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbcfd182-91e6-4abd-851e-a5b3462cc365" (UID: "cbcfd182-91e6-4abd-851e-a5b3462cc365"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:22:54.816135 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.816095 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbcfd182-91e6-4abd-851e-a5b3462cc365" (UID: "cbcfd182-91e6-4abd-851e-a5b3462cc365"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:22:54.816352 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.816200 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcfd182-91e6-4abd-851e-a5b3462cc365-kube-api-access-6cgp8" (OuterVolumeSpecName: "kube-api-access-6cgp8") pod "cbcfd182-91e6-4abd-851e-a5b3462cc365" (UID: "cbcfd182-91e6-4abd-851e-a5b3462cc365"). InnerVolumeSpecName "kube-api-access-6cgp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:22:54.914917 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.914881 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbcfd182-91e6-4abd-851e-a5b3462cc365-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:54.914917 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.914912 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgp8\" (UniqueName: \"kubernetes.io/projected/cbcfd182-91e6-4abd-851e-a5b3462cc365-kube-api-access-6cgp8\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:54.914917 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:54.914925 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfd182-91e6-4abd-851e-a5b3462cc365-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:22:55.616820 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:55.616792 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" Jul 10 16:22:55.617320 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:55.616792 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt" event={"ID":"cbcfd182-91e6-4abd-851e-a5b3462cc365","Type":"ContainerDied","Data":"d69cb95e06ebc4aecefd9fa389d3335682ce2f42c5054e52575f86ace3f8dfb4"} Jul 10 16:22:55.617320 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:55.616924 2576 scope.go:117] "RemoveContainer" containerID="05911917ffc95effe132cb5de387ce5dfd6466aa33b1cb5903b8f07b2bdb3068" Jul 10 16:22:55.633352 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:55.633326 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt"] Jul 10 16:22:55.638354 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:55.638330 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd511652-67b6fc59-4z2bt"] Jul 10 16:22:55.977081 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:55.977038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:22:57.496076 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:22:57.496042 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcfd182-91e6-4abd-851e-a5b3462cc365" path="/var/lib/kubelet/pods/cbcfd182-91e6-4abd-851e-a5b3462cc365/volumes" Jul 10 16:23:00.643434 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:00.643392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:00.741797 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:00.741763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:01.186728 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:01.186695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:01.279139 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:01.279108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:01.382462 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:01.382427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:02.773472 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:02.773433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:08.838014 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:08.837975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:08.902058 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:08.902025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:08.970017 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:08.969985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:09.068631 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:09.068599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:09.169413 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:09.169324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:09.592152 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:09.592112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:35.595060 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:35.594967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:41.516628 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:41.516586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:41.600874 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:41.600840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:41.669560 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:41.669522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:23:57.647819 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:23:57.647789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:03.613582 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:03.613548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:03.684490 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:03.684452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:03.752538 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:03.752498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:03.867839 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:03.867756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:14.356356 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:14.356319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:15.179801 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:15.179754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:21.358119 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:21.358082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:21.466621 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:21.466587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:21.563294 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:21.563263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:21.658774 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:21.658691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:22.460799 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:22.460767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:22.560740 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:22.560705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:22.662283 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:22.662223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:24.324313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:24.324278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:31.371097 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:31.371064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:31.470734 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:31.470692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:32.099333 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:32.099297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:38.878946 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:38.878907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:38.974041 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:38.974002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:39.105236 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.105194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:39.434451 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.434413 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb"] Jul 10 16:24:39.434878 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.434865 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcfd182-91e6-4abd-851e-a5b3462cc365" containerName="maas-api" Jul 10 16:24:39.434927 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.434881 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcfd182-91e6-4abd-851e-a5b3462cc365" containerName="maas-api" Jul 10 16:24:39.434962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.434949 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcfd182-91e6-4abd-851e-a5b3462cc365" containerName="maas-api" Jul 10 16:24:39.437983 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.437966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.440281 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.440230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79276d38-openshift-default-dockercfg-cdkxj\"" Jul 10 16:24:39.440509 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.440487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:39.462883 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.462848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb"] Jul 10 16:24:39.557646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-socket\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.557842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-data\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.557842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-certs\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.557842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-token\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.557842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-credential-socket\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.557842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwgg6\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-kube-api-access-rwgg6\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.557842 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istiod-ca-cert\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.558080 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-podinfo\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.558080 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.557869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-envoy\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.658890 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.658856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istiod-ca-cert\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659096 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.658904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-podinfo\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659096 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-envoy\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659096 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-socket\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659300 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-data\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659300 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-certs\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659300 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-token\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659300 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-credential-socket\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659300 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgg6\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-kube-api-access-rwgg6\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659686 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-data\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-certs\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istiod-ca-cert\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.659827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:39.660291 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.659909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-credential-socket\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.660291 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.660015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-socket\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.662106 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.662081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-envoy\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.662238 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.662220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-podinfo\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.668539 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.668512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgg6\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-kube-api-access-rwgg6\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.669023 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.668999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-token\") pod \"e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.748962 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.748869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:39.897297 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.897239 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb"] Jul 10 16:24:39.903019 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:24:39.900194 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce9fe0a_5826_45a7_935b_ecdabbcbe192.slice/crio-562fa54fd1881dbceace08eeef4465ecc8b9c8a02a92761aea2769506c45b3d5 WatchSource:0}: Error finding container 562fa54fd1881dbceace08eeef4465ecc8b9c8a02a92761aea2769506c45b3d5: Status 404 returned error can't find the container with id 562fa54fd1881dbceace08eeef4465ecc8b9c8a02a92761aea2769506c45b3d5 Jul 10 16:24:39.904789 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.904758 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:24:39.904899 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.904834 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:24:39.904899 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:39.904878 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:24:40.137031 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:40.136926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" event={"ID":"2ce9fe0a-5826-45a7-935b-ecdabbcbe192","Type":"ContainerStarted","Data":"382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9"} Jul 10 16:24:40.137206 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:40.137043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" event={"ID":"2ce9fe0a-5826-45a7-935b-ecdabbcbe192","Type":"ContainerStarted","Data":"562fa54fd1881dbceace08eeef4465ecc8b9c8a02a92761aea2769506c45b3d5"} Jul 10 16:24:40.161623 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:40.161564 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podStartSLOduration=1.161546905 podStartE2EDuration="1.161546905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:24:40.15897637 +0000 UTC m=+1769.229794434" watchObservedRunningTime="2026-07-10 16:24:40.161546905 +0000 UTC m=+1769.232364977" Jul 10 16:24:40.749562 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:40.749528 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:40.754531 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:40.754501 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:41.143306 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:41.143186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:41.144180 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:41.144160 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:42.235182 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.235148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:42.337840 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.337802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:42.348900 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.348864 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l"] Jul 10 16:24:42.352316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.352296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.354518 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.354491 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-79276d38-serving-cert\"" Jul 10 16:24:42.364903 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.364876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l"] Jul 10 16:24:42.382967 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.382933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c226cc-40ce-474e-8327-da54e155e370-openshift-service-ca\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.383145 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.382977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.383145 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.383004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2lf\" (UniqueName: \"kubernetes.io/projected/d3c226cc-40ce-474e-8327-da54e155e370-kube-api-access-9l2lf\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.451454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.451421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:42.483846 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.483816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c226cc-40ce-474e-8327-da54e155e370-openshift-service-ca\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.484083 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.483852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.484083 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.483877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2lf\" (UniqueName: \"kubernetes.io/projected/d3c226cc-40ce-474e-8327-da54e155e370-kube-api-access-9l2lf\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.484083 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:24:42.483992 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-serving-cert: secret "maas-api-e2e-mt-79276d38-serving-cert" not found Jul 10 16:24:42.484083 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:24:42.484070 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls podName:d3c226cc-40ce-474e-8327-da54e155e370 nodeName:}" failed. No retries permitted until 2026-07-10 16:24:42.984053033 +0000 UTC m=+1772.054871081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls") pod "maas-api-e2e-mt-79276d38-78586c9567-p4w5l" (UID: "d3c226cc-40ce-474e-8327-da54e155e370") : secret "maas-api-e2e-mt-79276d38-serving-cert" not found Jul 10 16:24:42.484631 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.484609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c226cc-40ce-474e-8327-da54e155e370-openshift-service-ca\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.501470 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.501397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2lf\" (UniqueName: \"kubernetes.io/projected/d3c226cc-40ce-474e-8327-da54e155e370-kube-api-access-9l2lf\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.562049 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.562017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:42.989297 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.989235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:42.992093 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:42.992054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls\") pod \"maas-api-e2e-mt-79276d38-78586c9567-p4w5l\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:43.143079 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:43.143044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:43.246599 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:43.246504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:43.264156 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:43.264127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:43.421598 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:43.421573 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l"] Jul 10 16:24:43.424491 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:24:43.424465 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c226cc_40ce_474e_8327_da54e155e370.slice/crio-dca692da1ef1ddc10d5cce5060ca2848fe48103230ccfe05886c93891a4d420e WatchSource:0}: Error finding container dca692da1ef1ddc10d5cce5060ca2848fe48103230ccfe05886c93891a4d420e: Status 404 returned error can't find the container with id dca692da1ef1ddc10d5cce5060ca2848fe48103230ccfe05886c93891a4d420e Jul 10 16:24:44.161637 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:44.161591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" event={"ID":"d3c226cc-40ce-474e-8327-da54e155e370","Type":"ContainerStarted","Data":"eaab3e7e0f1f81eed7aaa01ac87f3e80b9453d9db9d31c97cdcf6182c2efc201"} Jul 10 16:24:44.161826 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:44.161721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:44.161826 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:44.161742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" event={"ID":"d3c226cc-40ce-474e-8327-da54e155e370","Type":"ContainerStarted","Data":"dca692da1ef1ddc10d5cce5060ca2848fe48103230ccfe05886c93891a4d420e"} Jul 10 16:24:44.188561 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:44.188459 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" podStartSLOduration=1.664919243 podStartE2EDuration="2.188439349s" totalImagesPullingTime="523.520106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:24:44.188006351 +0000 UTC m=+1773.258824422" watchObservedRunningTime="2026-07-10 16:24:44.188439349 +0000 UTC m=+1773.259257411" Jul 10 16:24:46.746564 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:46.746530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:47.441752 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:47.441703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:47.543838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:47.543805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:47.893915 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:47.893828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:50.175531 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:50.175492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:54.043426 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:54.043380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:54.102277 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:54.102212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:54.467764 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:54.467712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:54.920611 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:54.920524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:55.019104 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.019068 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l"] Jul 10 16:24:55.019520 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.019446 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" podUID="d3c226cc-40ce-474e-8327-da54e155e370" containerName="maas-api" containerID="cri-o://eaab3e7e0f1f81eed7aaa01ac87f3e80b9453d9db9d31c97cdcf6182c2efc201" gracePeriod=30 Jul 10 16:24:55.059293 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.059241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:24:55.167237 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.167186 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" podUID="d3c226cc-40ce-474e-8327-da54e155e370" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.52:8443/health\": dial tcp 10.133.0.52:8443: connect: connection refused" Jul 10 16:24:55.238866 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.238835 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c226cc-40ce-474e-8327-da54e155e370" containerID="eaab3e7e0f1f81eed7aaa01ac87f3e80b9453d9db9d31c97cdcf6182c2efc201" exitCode=0 Jul 10 16:24:55.239037 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.238911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" event={"ID":"d3c226cc-40ce-474e-8327-da54e155e370","Type":"ContainerDied","Data":"eaab3e7e0f1f81eed7aaa01ac87f3e80b9453d9db9d31c97cdcf6182c2efc201"} Jul 10 16:24:55.301814 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.301791 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:55.410606 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.410569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c226cc-40ce-474e-8327-da54e155e370-openshift-service-ca\") pod \"d3c226cc-40ce-474e-8327-da54e155e370\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " Jul 10 16:24:55.410804 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.410656 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l2lf\" (UniqueName: \"kubernetes.io/projected/d3c226cc-40ce-474e-8327-da54e155e370-kube-api-access-9l2lf\") pod \"d3c226cc-40ce-474e-8327-da54e155e370\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " Jul 10 16:24:55.410804 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.410700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls\") pod \"d3c226cc-40ce-474e-8327-da54e155e370\" (UID: \"d3c226cc-40ce-474e-8327-da54e155e370\") " Jul 10 16:24:55.410981 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.410956 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c226cc-40ce-474e-8327-da54e155e370-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3c226cc-40ce-474e-8327-da54e155e370" (UID: "d3c226cc-40ce-474e-8327-da54e155e370"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:24:55.412930 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.412895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c226cc-40ce-474e-8327-da54e155e370-kube-api-access-9l2lf" (OuterVolumeSpecName: "kube-api-access-9l2lf") pod "d3c226cc-40ce-474e-8327-da54e155e370" (UID: "d3c226cc-40ce-474e-8327-da54e155e370"). InnerVolumeSpecName "kube-api-access-9l2lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:24:55.413037 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.412924 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3c226cc-40ce-474e-8327-da54e155e370" (UID: "d3c226cc-40ce-474e-8327-da54e155e370"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:24:55.512298 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.512274 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2lf\" (UniqueName: \"kubernetes.io/projected/d3c226cc-40ce-474e-8327-da54e155e370-kube-api-access-9l2lf\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:24:55.512419 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.512299 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c226cc-40ce-474e-8327-da54e155e370-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:24:55.512419 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:55.512328 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c226cc-40ce-474e-8327-da54e155e370-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:24:56.145546 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.145449 2576 patch_prober.go:28] interesting pod/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 16:24:56.145546 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.145516 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 16:24:56.245922 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.245824 2576 generic.go:356] "Generic (PLEG): container finished" podID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerID="382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9" exitCode=0 Jul 10 16:24:56.245922 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.245896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" event={"ID":"2ce9fe0a-5826-45a7-935b-ecdabbcbe192","Type":"ContainerDied","Data":"382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9"} Jul 10 16:24:56.246357 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.246340 2576 scope.go:117] "RemoveContainer" containerID="382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9" Jul 10 16:24:56.247314 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.247290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" event={"ID":"d3c226cc-40ce-474e-8327-da54e155e370","Type":"ContainerDied","Data":"dca692da1ef1ddc10d5cce5060ca2848fe48103230ccfe05886c93891a4d420e"} Jul 10 16:24:56.247389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.247316 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l" Jul 10 16:24:56.247389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.247352 2576 scope.go:117] "RemoveContainer" containerID="eaab3e7e0f1f81eed7aaa01ac87f3e80b9453d9db9d31c97cdcf6182c2efc201" Jul 10 16:24:56.247461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.247383 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:24:56.247498 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.247460 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:24:56.247538 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.247529 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:24:56.291129 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.291100 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l"] Jul 10 16:24:56.298444 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:56.298413 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79276d38-78586c9567-p4w5l"] Jul 10 16:24:57.254770 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:57.254739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" event={"ID":"2ce9fe0a-5826-45a7-935b-ecdabbcbe192","Type":"ContainerStarted","Data":"0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87"} Jul 10 16:24:57.499376 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:57.499338 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c226cc-40ce-474e-8327-da54e155e370" path="/var/lib/kubelet/pods/d3c226cc-40ce-474e-8327-da54e155e370/volumes" Jul 10 16:24:57.749324 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:57.749291 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:24:57.749560 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:57.749536 2576 patch_prober.go:28] interesting pod/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 16:24:57.749609 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:57.749596 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 16:24:58.750146 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:58.750115 2576 patch_prober.go:28] interesting pod/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 16:24:58.750566 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:58.750168 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 16:24:59.749673 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:59.749627 2576 patch_prober.go:28] interesting pod/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 16:24:59.749865 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:24:59.749693 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 16:25:00.750153 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:00.750112 2576 patch_prober.go:28] interesting pod/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 16:25:00.750697 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:00.750182 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 16:25:01.282609 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.282566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:01.348315 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.348284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:01.570418 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.570292 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb"] Jul 10 16:25:01.570622 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.570520 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" containerID="cri-o://0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87" gracePeriod=30 Jul 10 16:25:01.602393 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.602358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:01.777812 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.777774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:01.904592 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:01.904492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:02.004521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:02.004483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:02.142753 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:02.142722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:06.826522 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.826494 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:25:06.922813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922722 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-credential-socket\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.922813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwgg6\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-kube-api-access-rwgg6\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.922813 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-envoy\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922820 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-podinfo\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-certs\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-socket\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-data\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922949 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-token\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.922987 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istiod-ca-cert\") pod \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\" (UID: \"2ce9fe0a-5826-45a7-935b-ecdabbcbe192\") " Jul 10 16:25:06.923101 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923034 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:25:06.923515 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923140 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:25:06.923515 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923164 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:25:06.923515 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923347 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-credential-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:06.923515 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923367 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-certs\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:06.923515 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923381 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-workload-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:06.923515 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.923476 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:25:06.924168 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.924141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:25:06.925357 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.925333 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:25:06.925452 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.925427 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 16:25:06.925648 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.925630 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:25:06.925691 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:06.925640 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-kube-api-access-rwgg6" (OuterVolumeSpecName: "kube-api-access-rwgg6") pod "2ce9fe0a-5826-45a7-935b-ecdabbcbe192" (UID: "2ce9fe0a-5826-45a7-935b-ecdabbcbe192"). InnerVolumeSpecName "kube-api-access-rwgg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:25:07.023872 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.023832 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwgg6\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-kube-api-access-rwgg6\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:07.023872 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.023867 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-envoy\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:07.023872 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.023878 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-podinfo\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:07.023872 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.023885 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-data\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:07.024148 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.023892 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istio-token\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:07.024148 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.023901 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce9fe0a-5826-45a7-935b-ecdabbcbe192-istiod-ca-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:25:07.307737 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.307702 2576 generic.go:356] "Generic (PLEG): container finished" podID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerID="0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87" exitCode=0 Jul 10 16:25:07.307911 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.307743 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" Jul 10 16:25:07.307911 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.307785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" event={"ID":"2ce9fe0a-5826-45a7-935b-ecdabbcbe192","Type":"ContainerDied","Data":"0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87"} Jul 10 16:25:07.307911 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.307843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb" event={"ID":"2ce9fe0a-5826-45a7-935b-ecdabbcbe192","Type":"ContainerDied","Data":"562fa54fd1881dbceace08eeef4465ecc8b9c8a02a92761aea2769506c45b3d5"} Jul 10 16:25:07.307911 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.307868 2576 scope.go:117] "RemoveContainer" containerID="0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87" Jul 10 16:25:07.319650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.319627 2576 scope.go:117] "RemoveContainer" containerID="382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9" Jul 10 16:25:07.327935 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.327916 2576 scope.go:117] "RemoveContainer" containerID="0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87" Jul 10 16:25:07.328361 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:25:07.328327 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87\": container with ID starting with 0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87 not found: ID does not exist" containerID="0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87" Jul 10 16:25:07.328485 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.328371 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87"} err="failed to get container status \"0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87\": rpc error: code = NotFound desc = could not find container \"0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87\": container with ID starting with 0f42416e74fdcd2118b782ef9a7e1c72e7e3ddf10f071288fb09ccde1a849f87 not found: ID does not exist" Jul 10 16:25:07.328485 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.328395 2576 scope.go:117] "RemoveContainer" containerID="382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9" Jul 10 16:25:07.328740 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:25:07.328713 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9\": container with ID starting with 382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9 not found: ID does not exist" containerID="382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9" Jul 10 16:25:07.328821 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.328752 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9"} err="failed to get container status \"382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9\": rpc error: code = NotFound desc = could not find container \"382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9\": container with ID starting with 382689f88da6fbf70259a7c1dd38e3d92cf5b87cf5b7a4f9acd2f459398ea7f9 not found: ID does not exist" Jul 10 16:25:07.330713 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.330690 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb"] Jul 10 16:25:07.335926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.335907 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79276d38-openshift-default-7d766598b7-tdzcb"] Jul 10 16:25:07.495328 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:07.495294 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" path="/var/lib/kubelet/pods/2ce9fe0a-5826-45a7-935b-ecdabbcbe192/volumes" Jul 10 16:25:08.564930 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:08.564896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:08.669097 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:08.669060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:08.763894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:08.763860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:08.912338 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:08.912200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:11.483773 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:11.483744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:25:11.487302 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:11.487277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:25:11.490576 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:11.490555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:25:11.494692 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:11.494666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:25:15.211010 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:15.210978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:15.315445 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:15.315416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:15.416770 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:15.416734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:15.529804 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:15.529706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:22.112155 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:22.112122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:22.441409 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:22.441370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:22.773792 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:22.773681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:25.437766 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.437733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:25.543519 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.543481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:25.547935 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.547907 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx"] Jul 10 16:25:25.548338 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548324 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" Jul 10 16:25:25.548389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548339 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" Jul 10 16:25:25.548389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548358 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" Jul 10 16:25:25.548389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548365 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" Jul 10 16:25:25.548389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548374 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c226cc-40ce-474e-8327-da54e155e370" containerName="maas-api" Jul 10 16:25:25.548389 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548379 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c226cc-40ce-474e-8327-da54e155e370" containerName="maas-api" Jul 10 16:25:25.548544 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548433 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" Jul 10 16:25:25.548544 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.548442 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c226cc-40ce-474e-8327-da54e155e370" containerName="maas-api" Jul 10 16:25:25.552691 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.552672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.554738 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.554717 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-178db329-serving-cert\"" Jul 10 16:25:25.565825 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.565802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx"] Jul 10 16:25:25.594609 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.594576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.594823 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.594621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68540aae-3645-4d34-abce-77d5a6e91bf9-openshift-service-ca\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.594823 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.594716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfrg\" (UniqueName: \"kubernetes.io/projected/68540aae-3645-4d34-abce-77d5a6e91bf9-kube-api-access-ggfrg\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.637601 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.637556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:25.695861 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.695774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfrg\" (UniqueName: \"kubernetes.io/projected/68540aae-3645-4d34-abce-77d5a6e91bf9-kube-api-access-ggfrg\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.695861 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.695834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.696058 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.695880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68540aae-3645-4d34-abce-77d5a6e91bf9-openshift-service-ca\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.696058 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:25:25.695937 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-178db329-serving-cert: secret "maas-api-e2e-mt-178db329-serving-cert" not found Jul 10 16:25:25.696058 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:25:25.696017 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls podName:68540aae-3645-4d34-abce-77d5a6e91bf9 nodeName:}" failed. No retries permitted until 2026-07-10 16:25:26.196000875 +0000 UTC m=+1815.266818923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls") pod "maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" (UID: "68540aae-3645-4d34-abce-77d5a6e91bf9") : secret "maas-api-e2e-mt-178db329-serving-cert" not found Jul 10 16:25:25.696503 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.696486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68540aae-3645-4d34-abce-77d5a6e91bf9-openshift-service-ca\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.707391 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.707368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfrg\" (UniqueName: \"kubernetes.io/projected/68540aae-3645-4d34-abce-77d5a6e91bf9-kube-api-access-ggfrg\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:25.852625 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:25.852586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:26.122881 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:26.122789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:26.199541 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:26.199504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:26.202062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:26.202029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls\") pod \"maas-api-e2e-mt-178db329-6ff5875fcf-frxmx\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:26.222810 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:26.222774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:26.463406 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:26.463375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:26.606371 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:26.606342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx"] Jul 10 16:25:26.608706 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:25:26.608675 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68540aae_3645_4d34_abce_77d5a6e91bf9.slice/crio-d507b8ea4467d52bbfbef05f466b3010d2cb1811d98c359200c3cee03b2da8ea WatchSource:0}: Error finding container d507b8ea4467d52bbfbef05f466b3010d2cb1811d98c359200c3cee03b2da8ea: Status 404 returned error can't find the container with id d507b8ea4467d52bbfbef05f466b3010d2cb1811d98c359200c3cee03b2da8ea Jul 10 16:25:27.421039 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:27.420943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" event={"ID":"68540aae-3645-4d34-abce-77d5a6e91bf9","Type":"ContainerStarted","Data":"6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4"} Jul 10 16:25:27.421039 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:27.421000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" event={"ID":"68540aae-3645-4d34-abce-77d5a6e91bf9","Type":"ContainerStarted","Data":"d507b8ea4467d52bbfbef05f466b3010d2cb1811d98c359200c3cee03b2da8ea"} Jul 10 16:25:27.421289 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:27.421083 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:27.442473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:27.442422 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" podStartSLOduration=2.096802476 podStartE2EDuration="2.442403825s" totalImagesPullingTime="345.601349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:25:27.440747583 +0000 UTC m=+1816.511565674" watchObservedRunningTime="2026-07-10 16:25:27.442403825 +0000 UTC m=+1816.513221895" Jul 10 16:25:29.974646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:29.974606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:25:33.431930 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:33.431896 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:25:39.289206 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:25:39.289157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:02.427775 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:02.427741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:02.829859 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:02.829818 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx"] Jul 10 16:26:02.830144 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:02.830095 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" podUID="68540aae-3645-4d34-abce-77d5a6e91bf9" containerName="maas-api" containerID="cri-o://6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4" gracePeriod=30 Jul 10 16:26:02.853818 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:02.853791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:03.090799 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.090731 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:26:03.233789 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.233744 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls\") pod \"68540aae-3645-4d34-abce-77d5a6e91bf9\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " Jul 10 16:26:03.233789 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.233782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggfrg\" (UniqueName: \"kubernetes.io/projected/68540aae-3645-4d34-abce-77d5a6e91bf9-kube-api-access-ggfrg\") pod \"68540aae-3645-4d34-abce-77d5a6e91bf9\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " Jul 10 16:26:03.234052 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.233838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68540aae-3645-4d34-abce-77d5a6e91bf9-openshift-service-ca\") pod \"68540aae-3645-4d34-abce-77d5a6e91bf9\" (UID: \"68540aae-3645-4d34-abce-77d5a6e91bf9\") " Jul 10 16:26:03.234363 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.234327 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68540aae-3645-4d34-abce-77d5a6e91bf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68540aae-3645-4d34-abce-77d5a6e91bf9" (UID: "68540aae-3645-4d34-abce-77d5a6e91bf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:26:03.236003 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.235978 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68540aae-3645-4d34-abce-77d5a6e91bf9" (UID: "68540aae-3645-4d34-abce-77d5a6e91bf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:26:03.236097 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.236016 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68540aae-3645-4d34-abce-77d5a6e91bf9-kube-api-access-ggfrg" (OuterVolumeSpecName: "kube-api-access-ggfrg") pod "68540aae-3645-4d34-abce-77d5a6e91bf9" (UID: "68540aae-3645-4d34-abce-77d5a6e91bf9"). InnerVolumeSpecName "kube-api-access-ggfrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:26:03.335324 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.335289 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68540aae-3645-4d34-abce-77d5a6e91bf9-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:26:03.335324 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.335321 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68540aae-3645-4d34-abce-77d5a6e91bf9-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:26:03.335324 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.335330 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggfrg\" (UniqueName: \"kubernetes.io/projected/68540aae-3645-4d34-abce-77d5a6e91bf9-kube-api-access-ggfrg\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:26:03.606093 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.606057 2576 generic.go:356] "Generic (PLEG): container finished" podID="68540aae-3645-4d34-abce-77d5a6e91bf9" containerID="6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4" exitCode=0 Jul 10 16:26:03.606093 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.606102 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" Jul 10 16:26:03.606577 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.606144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" event={"ID":"68540aae-3645-4d34-abce-77d5a6e91bf9","Type":"ContainerDied","Data":"6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4"} Jul 10 16:26:03.606577 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.606202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx" event={"ID":"68540aae-3645-4d34-abce-77d5a6e91bf9","Type":"ContainerDied","Data":"d507b8ea4467d52bbfbef05f466b3010d2cb1811d98c359200c3cee03b2da8ea"} Jul 10 16:26:03.606577 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.606224 2576 scope.go:117] "RemoveContainer" containerID="6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4" Jul 10 16:26:03.617037 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.617018 2576 scope.go:117] "RemoveContainer" containerID="6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4" Jul 10 16:26:03.617357 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:26:03.617330 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4\": container with ID starting with 6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4 not found: ID does not exist" containerID="6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4" Jul 10 16:26:03.617419 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.617369 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4"} err="failed to get container status \"6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4\": rpc error: code = NotFound desc = could not find container \"6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4\": container with ID starting with 6d01d04321420b56984cf2207013c19980b3c7c6824bc71d78e21e780b3faee4 not found: ID does not exist" Jul 10 16:26:03.621860 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.621837 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx"] Jul 10 16:26:03.626456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:03.626433 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-178db329-6ff5875fcf-frxmx"] Jul 10 16:26:05.495163 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:05.495116 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68540aae-3645-4d34-abce-77d5a6e91bf9" path="/var/lib/kubelet/pods/68540aae-3645-4d34-abce-77d5a6e91bf9/volumes" Jul 10 16:26:08.587678 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:08.587641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:13.664635 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:13.664600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:23.146424 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:23.146386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:23.215548 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:23.215511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:53.996925 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:53.996885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:54.072169 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:54.072132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:26:58.737625 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:26:58.737593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:00.210894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:00.210859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:06.167048 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:06.166993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:06.265897 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:06.265857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:06.398612 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:06.398574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:06.633849 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:06.633814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:07.714388 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:07.714349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:07.810609 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:07.810576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:07.911456 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:07.911423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:08.726523 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:08.726483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:08.829857 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:08.829802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:08.927836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:08.927787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:10.372561 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:10.372526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:16.336644 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:16.336609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:16.401779 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:16.401705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:17.017086 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:17.016998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:22.924464 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:22.924427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:23.028805 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:23.028773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:23.118779 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:23.118743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:33.666229 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:33.666195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:34.074699 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:34.074663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:27:55.020004 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:27:55.019969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:00.823184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:00.823148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:00.923650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:00.923618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:01.181700 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.181617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:01.477307 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.477273 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq"] Jul 10 16:28:01.477755 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.477741 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68540aae-3645-4d34-abce-77d5a6e91bf9" containerName="maas-api" Jul 10 16:28:01.477801 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.477758 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="68540aae-3645-4d34-abce-77d5a6e91bf9" containerName="maas-api" Jul 10 16:28:01.477843 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.477834 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce9fe0a-5826-45a7-935b-ecdabbcbe192" containerName="istio-proxy" Jul 10 16:28:01.477879 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.477844 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="68540aae-3645-4d34-abce-77d5a6e91bf9" containerName="maas-api" Jul 10 16:28:01.480895 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.480879 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.482827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.482811 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0a40f1-openshift-default-dockercfg-5mrqm\"" Jul 10 16:28:01.496120 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.496093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq"] Jul 10 16:28:01.499192 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.499151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:01.597414 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-data\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597414 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xv67\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-kube-api-access-8xv67\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-socket\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-podinfo\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-token\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istiod-ca-cert\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597835 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-credential-socket\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597835 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-envoy\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.597933 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.597832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-certs\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699054 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-certs\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699232 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-data\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699232 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xv67\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-kube-api-access-8xv67\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699232 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-socket\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-podinfo\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-token\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istiod-ca-cert\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-credential-socket\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699463 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-envoy\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699721 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-data\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-socket\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-certs\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.699924 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.699834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-credential-socket\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.700203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.700185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istiod-ca-cert\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.701838 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.701809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-envoy\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.701918 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.701892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-podinfo\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.706373 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.706343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-token\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.706587 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.706570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xv67\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-kube-api-access-8xv67\") pod \"e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.793659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.793564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:01.937170 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.937142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq"] Jul 10 16:28:01.939468 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:28:01.939437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1b7c02_acf0_4f04_848a_299aeb1c8f86.slice/crio-9a26f545aa9264229e3520c7e43df026f8d0fb2a77b8f0d108d62ce839c37d2f WatchSource:0}: Error finding container 9a26f545aa9264229e3520c7e43df026f8d0fb2a77b8f0d108d62ce839c37d2f: Status 404 returned error can't find the container with id 9a26f545aa9264229e3520c7e43df026f8d0fb2a77b8f0d108d62ce839c37d2f Jul 10 16:28:01.941706 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.941673 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:28:01.941840 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.941758 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:28:01.941840 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:01.941803 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:28:02.122483 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:02.122382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:02.207735 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:02.207700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" event={"ID":"ab1b7c02-acf0-4f04-848a-299aeb1c8f86","Type":"ContainerStarted","Data":"7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582"} Jul 10 16:28:02.207894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:02.207763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" event={"ID":"ab1b7c02-acf0-4f04-848a-299aeb1c8f86","Type":"ContainerStarted","Data":"9a26f545aa9264229e3520c7e43df026f8d0fb2a77b8f0d108d62ce839c37d2f"} Jul 10 16:28:02.225529 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:02.225482 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podStartSLOduration=1.225465862 podStartE2EDuration="1.225465862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:28:02.224043302 +0000 UTC m=+1971.294861372" watchObservedRunningTime="2026-07-10 16:28:02.225465862 +0000 UTC m=+1971.296283932" Jul 10 16:28:02.794641 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:02.794605 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:03.799292 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:03.799260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:04.218194 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.218162 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:04.219568 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.219537 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:04.244764 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.244725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:04.344043 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.344010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:04.459948 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.459914 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7"] Jul 10 16:28:04.463730 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.463709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.465866 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.465841 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0a40f1-serving-cert\"" Jul 10 16:28:04.470955 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.470892 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7"] Jul 10 16:28:04.500316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.500280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:04.628998 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.628958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqzm\" (UniqueName: \"kubernetes.io/projected/60693b85-50bc-41d3-b1b7-139c5e261009-kube-api-access-hkqzm\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.628998 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.629004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60693b85-50bc-41d3-b1b7-139c5e261009-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.629357 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.629092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.730435 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.730332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.730435 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.730398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqzm\" (UniqueName: \"kubernetes.io/projected/60693b85-50bc-41d3-b1b7-139c5e261009-kube-api-access-hkqzm\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.730435 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.730422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60693b85-50bc-41d3-b1b7-139c5e261009-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.730721 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:28:04.730491 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-serving-cert: secret "maas-api-e2e-inf-a-0a40f1-serving-cert" not found Jul 10 16:28:04.730721 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:28:04.730559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls podName:60693b85-50bc-41d3-b1b7-139c5e261009 nodeName:}" failed. No retries permitted until 2026-07-10 16:28:05.230542857 +0000 UTC m=+1974.301360906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls") pod "maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" (UID: "60693b85-50bc-41d3-b1b7-139c5e261009") : secret "maas-api-e2e-inf-a-0a40f1-serving-cert" not found Jul 10 16:28:04.731049 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.731032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60693b85-50bc-41d3-b1b7-139c5e261009-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.740830 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.740804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqzm\" (UniqueName: \"kubernetes.io/projected/60693b85-50bc-41d3-b1b7-139c5e261009-kube-api-access-hkqzm\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:04.788815 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.788784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:04.942416 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:04.942379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:05.043277 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:05.043163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:05.235951 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:05.235913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:05.238558 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:05.238535 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:05.377374 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:05.377281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:05.524408 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:05.524380 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7"] Jul 10 16:28:05.526558 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:28:05.526524 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60693b85_50bc_41d3_b1b7_139c5e261009.slice/crio-78aa26ff0e1527a4b3515fccdeac97aef340e03775d4fdaa57afe440ffd2d943 WatchSource:0}: Error finding container 78aa26ff0e1527a4b3515fccdeac97aef340e03775d4fdaa57afe440ffd2d943: Status 404 returned error can't find the container with id 78aa26ff0e1527a4b3515fccdeac97aef340e03775d4fdaa57afe440ffd2d943 Jul 10 16:28:05.527988 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:05.527968 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:28:06.228124 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:06.228086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" event={"ID":"60693b85-50bc-41d3-b1b7-139c5e261009","Type":"ContainerStarted","Data":"15c81c2c6678dacf134978e4ea1ec978040267f97b797a9c7e0e772c65dc774b"} Jul 10 16:28:06.228606 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:06.228138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" event={"ID":"60693b85-50bc-41d3-b1b7-139c5e261009","Type":"ContainerStarted","Data":"78aa26ff0e1527a4b3515fccdeac97aef340e03775d4fdaa57afe440ffd2d943"} Jul 10 16:28:06.244905 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:06.244853 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" podStartSLOduration=1.7521229699999998 podStartE2EDuration="2.24483759s" totalImagesPullingTime="492.71462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:28:06.243324659 +0000 UTC m=+1975.314142745" watchObservedRunningTime="2026-07-10 16:28:06.24483759 +0000 UTC m=+1975.315655660" Jul 10 16:28:07.233130 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:07.233103 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:08.682390 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:08.682354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:09.018188 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:09.018105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:09.339543 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:09.339467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:09.439782 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:09.439752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:12.192415 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:12.192381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:12.267539 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:12.267509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:12.332159 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:12.332125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:12.445293 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:12.445187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:12.775439 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:12.775344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:12.877232 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:12.877195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:13.243233 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:13.243204 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:28:16.726676 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:16.726632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:19.042944 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:19.042904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:19.683829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:19.682937 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:36202->10.133.0.54:15021: read: connection reset by peer" start-of-body= Jul 10 16:28:19.683829 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:19.683000 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:36202->10.133.0.54:15021: read: connection reset by peer" Jul 10 16:28:20.304033 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:20.304000 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerID="7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582" exitCode=0 Jul 10 16:28:20.304445 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:20.304062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" event={"ID":"ab1b7c02-acf0-4f04-848a-299aeb1c8f86","Type":"ContainerDied","Data":"7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582"} Jul 10 16:28:20.304489 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:20.304455 2576 scope.go:117] "RemoveContainer" containerID="7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582" Jul 10 16:28:20.305537 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:20.305511 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:28:20.305607 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:20.305587 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:28:20.305648 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:20.305621 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:28:21.311657 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:21.311622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" event={"ID":"ab1b7c02-acf0-4f04-848a-299aeb1c8f86","Type":"ContainerStarted","Data":"4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399"} Jul 10 16:28:21.794401 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:21.794368 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:21.794786 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:21.794700 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 16:28:21.794947 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:21.794925 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 16:28:22.794831 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:22.794784 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 16:28:22.795366 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:22.794858 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 16:28:23.794659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:23.794609 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 16:28:23.794878 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:23.794703 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 16:28:24.794613 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:24.794573 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 16:28:24.794941 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:24.794909 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 16:28:25.779683 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:25.779627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:25.797672 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:25.797643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:25.798126 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:25.798102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:25.798880 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:25.798866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:28:25.837747 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:25.837711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:33.729379 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:33.729342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:33.830651 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:33.830610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:46.049835 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:46.049791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:28:46.150678 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:28:46.150644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:02.048960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:02.048920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:20.539199 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:20.539159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:21.519872 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:21.519834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:21.638159 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:21.638113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:22.157907 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.157875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:22.258585 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.258551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:22.359162 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.359129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:22.476184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.476149 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7"] Jul 10 16:29:22.476430 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.476401 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" podUID="60693b85-50bc-41d3-b1b7-139c5e261009" containerName="maas-api" containerID="cri-o://15c81c2c6678dacf134978e4ea1ec978040267f97b797a9c7e0e772c65dc774b" gracePeriod=30 Jul 10 16:29:22.498919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.498895 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:22.627124 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.627088 2576 generic.go:356] "Generic (PLEG): container finished" podID="60693b85-50bc-41d3-b1b7-139c5e261009" containerID="15c81c2c6678dacf134978e4ea1ec978040267f97b797a9c7e0e772c65dc774b" exitCode=0 Jul 10 16:29:22.627339 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.627138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" event={"ID":"60693b85-50bc-41d3-b1b7-139c5e261009","Type":"ContainerDied","Data":"15c81c2c6678dacf134978e4ea1ec978040267f97b797a9c7e0e772c65dc774b"} Jul 10 16:29:22.738613 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.738548 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:29:22.862180 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.862147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls\") pod \"60693b85-50bc-41d3-b1b7-139c5e261009\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " Jul 10 16:29:22.862381 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.862217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60693b85-50bc-41d3-b1b7-139c5e261009-openshift-service-ca\") pod \"60693b85-50bc-41d3-b1b7-139c5e261009\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " Jul 10 16:29:22.862381 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.862312 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkqzm\" (UniqueName: \"kubernetes.io/projected/60693b85-50bc-41d3-b1b7-139c5e261009-kube-api-access-hkqzm\") pod \"60693b85-50bc-41d3-b1b7-139c5e261009\" (UID: \"60693b85-50bc-41d3-b1b7-139c5e261009\") " Jul 10 16:29:22.862658 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.862629 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60693b85-50bc-41d3-b1b7-139c5e261009-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60693b85-50bc-41d3-b1b7-139c5e261009" (UID: "60693b85-50bc-41d3-b1b7-139c5e261009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:29:22.864517 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.864495 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60693b85-50bc-41d3-b1b7-139c5e261009-kube-api-access-hkqzm" (OuterVolumeSpecName: "kube-api-access-hkqzm") pod "60693b85-50bc-41d3-b1b7-139c5e261009" (UID: "60693b85-50bc-41d3-b1b7-139c5e261009"). InnerVolumeSpecName "kube-api-access-hkqzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:29:22.864605 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.864542 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60693b85-50bc-41d3-b1b7-139c5e261009" (UID: "60693b85-50bc-41d3-b1b7-139c5e261009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:29:22.963703 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.963666 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkqzm\" (UniqueName: \"kubernetes.io/projected/60693b85-50bc-41d3-b1b7-139c5e261009-kube-api-access-hkqzm\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:29:22.963703 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.963698 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60693b85-50bc-41d3-b1b7-139c5e261009-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:29:22.963703 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:22.963708 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60693b85-50bc-41d3-b1b7-139c5e261009-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:29:23.633606 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:23.633574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" event={"ID":"60693b85-50bc-41d3-b1b7-139c5e261009","Type":"ContainerDied","Data":"78aa26ff0e1527a4b3515fccdeac97aef340e03775d4fdaa57afe440ffd2d943"} Jul 10 16:29:23.633787 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:23.633632 2576 scope.go:117] "RemoveContainer" containerID="15c81c2c6678dacf134978e4ea1ec978040267f97b797a9c7e0e772c65dc774b" Jul 10 16:29:23.633787 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:23.633589 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7" Jul 10 16:29:23.649005 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:23.648978 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7"] Jul 10 16:29:23.652376 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:23.652351 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a40f1-6854696965-mqqb7"] Jul 10 16:29:25.497399 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:25.497360 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60693b85-50bc-41d3-b1b7-139c5e261009" path="/var/lib/kubelet/pods/60693b85-50bc-41d3-b1b7-139c5e261009/volumes" Jul 10 16:29:27.244580 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:27.244545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:29:51.492976 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:29:51.492905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:00.142026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.141978 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck"] Jul 10 16:30:00.142644 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.142604 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60693b85-50bc-41d3-b1b7-139c5e261009" containerName="maas-api" Jul 10 16:30:00.142644 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.142625 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60693b85-50bc-41d3-b1b7-139c5e261009" containerName="maas-api" Jul 10 16:30:00.142773 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.142737 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="60693b85-50bc-41d3-b1b7-139c5e261009" containerName="maas-api" Jul 10 16:30:00.145527 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.145506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" Jul 10 16:30:00.147687 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.147666 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rf26q\"" Jul 10 16:30:00.151599 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.151574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck"] Jul 10 16:30:00.157016 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.156995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" Jul 10 16:30:00.299667 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.299641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck"] Jul 10 16:30:00.301815 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:30:00.301780 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb3b483_da08_4907_a864_76d59dfe7544.slice/crio-f7bae7c7882690fcc8c3cddf3c4d4b5ff7b932eaf8d4cbf49b6110aff875793f WatchSource:0}: Error finding container f7bae7c7882690fcc8c3cddf3c4d4b5ff7b932eaf8d4cbf49b6110aff875793f: Status 404 returned error can't find the container with id f7bae7c7882690fcc8c3cddf3c4d4b5ff7b932eaf8d4cbf49b6110aff875793f Jul 10 16:30:00.826219 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.826123 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" event={"ID":"bdb3b483-da08-4907-a864-76d59dfe7544","Type":"ContainerStarted","Data":"7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163"} Jul 10 16:30:00.826219 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.826178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" event={"ID":"bdb3b483-da08-4907-a864-76d59dfe7544","Type":"ContainerStarted","Data":"f7bae7c7882690fcc8c3cddf3c4d4b5ff7b932eaf8d4cbf49b6110aff875793f"} Jul 10 16:30:00.842960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:00.842915 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" podStartSLOduration=0.84289853 podStartE2EDuration="842.89853ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:30:00.84122367 +0000 UTC m=+2089.912041760" watchObservedRunningTime="2026-07-10 16:30:00.84289853 +0000 UTC m=+2089.913716639" Jul 10 16:30:04.258312 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:04.258274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:04.506505 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:04.506463 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq"] Jul 10 16:30:04.506756 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:04.506728 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" containerID="cri-o://4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399" gracePeriod=30 Jul 10 16:30:04.526681 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:04.526588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:04.766819 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:04.766783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:04.931436 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:04.931339 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:05.351769 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:05.351733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:05.582073 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:05.582041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:05.681936 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:05.681851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:05.763831 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:05.763792 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck"] Jul 10 16:30:05.763997 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:05.763969 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" podUID="bdb3b483-da08-4907-a864-76d59dfe7544" containerName="cleanup" containerID="cri-o://7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163" gracePeriod=30 Jul 10 16:30:05.766534 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:05.766498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:09.766702 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.766641 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:30:09.872591 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.872556 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerID="4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399" exitCode=0 Jul 10 16:30:09.872803 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.872609 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" Jul 10 16:30:09.872803 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.872635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" event={"ID":"ab1b7c02-acf0-4f04-848a-299aeb1c8f86","Type":"ContainerDied","Data":"4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399"} Jul 10 16:30:09.872803 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.872704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq" event={"ID":"ab1b7c02-acf0-4f04-848a-299aeb1c8f86","Type":"ContainerDied","Data":"9a26f545aa9264229e3520c7e43df026f8d0fb2a77b8f0d108d62ce839c37d2f"} Jul 10 16:30:09.872803 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.872734 2576 scope.go:117] "RemoveContainer" containerID="4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399" Jul 10 16:30:09.882373 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.882352 2576 scope.go:117] "RemoveContainer" containerID="7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582" Jul 10 16:30:09.890969 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.890947 2576 scope.go:117] "RemoveContainer" containerID="4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399" Jul 10 16:30:09.891238 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:30:09.891218 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399\": container with ID starting with 4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399 not found: ID does not exist" containerID="4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399" Jul 10 16:30:09.891316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.891247 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399"} err="failed to get container status \"4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399\": rpc error: code = NotFound desc = could not find container \"4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399\": container with ID starting with 4d5992d079dedc58a800f0258bfce624ff701bcfe84e33b9e409090c22cfb399 not found: ID does not exist" Jul 10 16:30:09.891316 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.891290 2576 scope.go:117] "RemoveContainer" containerID="7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582" Jul 10 16:30:09.891544 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:30:09.891526 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582\": container with ID starting with 7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582 not found: ID does not exist" containerID="7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582" Jul 10 16:30:09.891587 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.891552 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582"} err="failed to get container status \"7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582\": rpc error: code = NotFound desc = could not find container \"7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582\": container with ID starting with 7d605c88dace7fdb7a5b1a9cbe7c48f5e5b620cc94fb9fa8f7016a45c9501582 not found: ID does not exist" Jul 10 16:30:09.905887 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.905862 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-token\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.905977 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.905906 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istiod-ca-cert\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.905977 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.905932 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-envoy\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.905977 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.905968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-podinfo\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.906105 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906020 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-credential-socket\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.906105 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-certs\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.906211 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906156 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-data\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.906211 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xv67\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-kube-api-access-8xv67\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.906340 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906231 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-socket\") pod \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\" (UID: \"ab1b7c02-acf0-4f04-848a-299aeb1c8f86\") " Jul 10 16:30:09.906394 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906361 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:09.906394 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906377 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:09.906507 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906397 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:30:09.906611 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906580 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:09.906872 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906846 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:09.907131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906873 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istiod-ca-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:09.907131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906888 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-credential-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:09.907131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.906902 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-workload-certs\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:09.907131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.907014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:09.908398 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.908368 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:09.908579 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.908561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-kube-api-access-8xv67" (OuterVolumeSpecName: "kube-api-access-8xv67") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "kube-api-access-8xv67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:30:09.908746 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.908733 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:30:09.908849 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:09.908832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab1b7c02-acf0-4f04-848a-299aeb1c8f86" (UID: "ab1b7c02-acf0-4f04-848a-299aeb1c8f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 16:30:10.007823 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.007784 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-data\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:10.007823 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.007819 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xv67\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-kube-api-access-8xv67\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:10.007823 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.007829 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-token\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:10.008069 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.007837 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-envoy\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:10.008069 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.007846 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b7c02-acf0-4f04-848a-299aeb1c8f86-istio-podinfo\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:10.195812 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.195784 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq"] Jul 10 16:30:10.201394 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:10.201367 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0a40f1-openshift-default-649967c4bf-p2tgq"] Jul 10 16:30:11.322973 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.322937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:11.412815 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.412780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:11.504881 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.504843 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" path="/var/lib/kubelet/pods/ab1b7c02-acf0-4f04-848a-299aeb1c8f86/volumes" Jul 10 16:30:11.520378 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.520350 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:30:11.523336 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.523316 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:30:11.524016 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.523998 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:30:11.526824 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.526806 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:30:11.597374 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.597292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:11.827845 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:11.827810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:12.162667 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.162635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:12.450841 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.450810 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww"] Jul 10 16:30:12.451320 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.451303 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" Jul 10 16:30:12.451391 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.451322 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" Jul 10 16:30:12.451391 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.451336 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" Jul 10 16:30:12.451391 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.451345 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" Jul 10 16:30:12.451550 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.451450 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" Jul 10 16:30:12.451550 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.451460 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1b7c02-acf0-4f04-848a-299aeb1c8f86" containerName="istio-proxy" Jul 10 16:30:12.456164 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.456138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.458473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.458452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2b206a-openshift-default-dockercfg-kk4gd\"" Jul 10 16:30:12.464488 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.464466 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww"] Jul 10 16:30:12.478166 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.478138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:12.631234 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-data\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631234 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-token\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631497 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dw7\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-kube-api-access-v2dw7\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631497 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-podinfo\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631497 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istiod-ca-cert\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631497 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-credential-socket\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631497 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-socket\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631497 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-certs\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.631717 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.631505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-envoy\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.704454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.704376 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:12.732601 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-credential-socket\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.732792 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-socket\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.732792 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-certs\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.732792 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-envoy\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.732792 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-data\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733014 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-token\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733014 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dw7\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-kube-api-access-v2dw7\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733014 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-podinfo\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733014 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.732944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istiod-ca-cert\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733210 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.733089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-credential-socket\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733210 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.733108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-socket\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733210 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.733138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-certs\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.733390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-data\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.733650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.733630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istiod-ca-cert\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.735369 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.735347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-envoy\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.735543 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.735524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-podinfo\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.740562 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.740538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-token\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.740650 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.740596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dw7\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-kube-api-access-v2dw7\") pod \"e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.767758 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.767715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:12.907583 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.907553 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww"] Jul 10 16:30:12.909830 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:30:12.909806 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc002e2d_1dbc_4a36_89c5_85e3a062e534.slice/crio-7afffc2372c5c73850133ba4076ed954331d5ad137505c706799f60d734dab1e WatchSource:0}: Error finding container 7afffc2372c5c73850133ba4076ed954331d5ad137505c706799f60d734dab1e: Status 404 returned error can't find the container with id 7afffc2372c5c73850133ba4076ed954331d5ad137505c706799f60d734dab1e Jul 10 16:30:12.912358 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.912330 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:12.912428 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.912391 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:12.912466 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:12.912426 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:13.899959 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:13.899920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" event={"ID":"fc002e2d-1dbc-4a36-89c5-85e3a062e534","Type":"ContainerStarted","Data":"7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb"} Jul 10 16:30:13.900367 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:13.899977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" event={"ID":"fc002e2d-1dbc-4a36-89c5-85e3a062e534","Type":"ContainerStarted","Data":"7afffc2372c5c73850133ba4076ed954331d5ad137505c706799f60d734dab1e"} Jul 10 16:30:13.920884 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:13.920832 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podStartSLOduration=1.920815887 podStartE2EDuration="1.920815887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:30:13.918858455 +0000 UTC m=+2102.989676524" watchObservedRunningTime="2026-07-10 16:30:13.920815887 +0000 UTC m=+2102.991633957" Jul 10 16:30:14.768914 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:14.768874 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:14.774134 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:14.774104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:14.905765 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:14.905726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:14.906640 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:14.906617 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:15.850205 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:15.850134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:15.957298 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:15.957244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:16.314806 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:16.314775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:16.415935 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:16.415906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:16.779003 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:16.778970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:17.071868 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:17.071761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:17.244634 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:17.244602 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:17.311686 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:17.311649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:20.364473 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:20.364438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:20.835712 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:20.835673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:21.159863 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:21.159788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:21.336325 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:21.336292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:23.871992 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:23.871929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:23.945439 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:23.945403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:23.946642 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:23.946614 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp"] Jul 10 16:30:23.950563 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:23.950538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:23.952636 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:23.952611 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1375a2-serving-cert\"" Jul 10 16:30:23.959678 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:23.959095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp"] Jul 10 16:30:24.006535 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.006503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:24.032827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.032791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvmb\" (UniqueName: \"kubernetes.io/projected/c5a56517-3924-4a64-9c3d-5c375d269ed7-kube-api-access-8vvmb\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.032975 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.032848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a56517-3924-4a64-9c3d-5c375d269ed7-openshift-service-ca\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.032975 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.032887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a56517-3924-4a64-9c3d-5c375d269ed7-maas-api-tls\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.106384 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.106341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:24.133676 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.133587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvmb\" (UniqueName: \"kubernetes.io/projected/c5a56517-3924-4a64-9c3d-5c375d269ed7-kube-api-access-8vvmb\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.133676 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.133635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a56517-3924-4a64-9c3d-5c375d269ed7-openshift-service-ca\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.133676 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.133660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a56517-3924-4a64-9c3d-5c375d269ed7-maas-api-tls\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.134363 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.134327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a56517-3924-4a64-9c3d-5c375d269ed7-openshift-service-ca\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.136233 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.136206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a56517-3924-4a64-9c3d-5c375d269ed7-maas-api-tls\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.141340 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.141321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvmb\" (UniqueName: \"kubernetes.io/projected/c5a56517-3924-4a64-9c3d-5c375d269ed7-kube-api-access-8vvmb\") pod \"maas-api-e2e-api-b-1375a2-576646fb96-xn8bp\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.264487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.264441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:24.404841 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.404809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp"] Jul 10 16:30:24.406210 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:30:24.406180 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a56517_3924_4a64_9c3d_5c375d269ed7.slice/crio-46f52962fde202d175973e648798020bf20e773f6c08c59c321b0471426c4e96 WatchSource:0}: Error finding container 46f52962fde202d175973e648798020bf20e773f6c08c59c321b0471426c4e96: Status 404 returned error can't find the container with id 46f52962fde202d175973e648798020bf20e773f6c08c59c321b0471426c4e96 Jul 10 16:30:24.542602 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.542561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:24.643026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.642990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:24.957267 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:24.957225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" event={"ID":"c5a56517-3924-4a64-9c3d-5c375d269ed7","Type":"ContainerStarted","Data":"46f52962fde202d175973e648798020bf20e773f6c08c59c321b0471426c4e96"} Jul 10 16:30:25.964607 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:25.964567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" event={"ID":"c5a56517-3924-4a64-9c3d-5c375d269ed7","Type":"ContainerStarted","Data":"81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb"} Jul 10 16:30:25.964998 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:25.964698 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:25.984194 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:25.984115 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" podStartSLOduration=2.375252892 podStartE2EDuration="2.984093498s" totalImagesPullingTime="608.840606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:30:25.982770389 +0000 UTC m=+2115.053588459" watchObservedRunningTime="2026-07-10 16:30:25.984093498 +0000 UTC m=+2115.054911570" Jul 10 16:30:28.367546 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:28.367514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:30.946478 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:30.946437 2576 patch_prober.go:28] interesting pod/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 16:30:30.946938 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:30.946513 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 16:30:31.976409 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:31.976378 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:33.008629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:33.008519 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerID="7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb" exitCode=0 Jul 10 16:30:33.008629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:33.008598 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" event={"ID":"fc002e2d-1dbc-4a36-89c5-85e3a062e534","Type":"ContainerDied","Data":"7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb"} Jul 10 16:30:33.009114 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:33.009098 2576 scope.go:117] "RemoveContainer" containerID="7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb" Jul 10 16:30:33.010140 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:33.010113 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:33.010191 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:33.010180 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:33.010231 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:33.010213 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:34.015387 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:34.015348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" event={"ID":"fc002e2d-1dbc-4a36-89c5-85e3a062e534","Type":"ContainerStarted","Data":"2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628"} Jul 10 16:30:34.768213 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:34.768173 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:34.768478 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:34.768455 2576 patch_prober.go:28] interesting pod/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 16:30:34.768566 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:34.768532 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 16:30:35.768516 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:35.768479 2576 patch_prober.go:28] interesting pod/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 16:30:35.768772 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:35.768557 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 16:30:36.006868 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.006841 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" Jul 10 16:30:36.027539 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.027450 2576 generic.go:356] "Generic (PLEG): container finished" podID="bdb3b483-da08-4907-a864-76d59dfe7544" containerID="7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163" exitCode=137 Jul 10 16:30:36.027539 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.027491 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" Jul 10 16:30:36.027539 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.027503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" event={"ID":"bdb3b483-da08-4907-a864-76d59dfe7544","Type":"ContainerDied","Data":"7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163"} Jul 10 16:30:36.027798 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.027562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck" event={"ID":"bdb3b483-da08-4907-a864-76d59dfe7544","Type":"ContainerDied","Data":"f7bae7c7882690fcc8c3cddf3c4d4b5ff7b932eaf8d4cbf49b6110aff875793f"} Jul 10 16:30:36.027798 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.027593 2576 scope.go:117] "RemoveContainer" containerID="7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163" Jul 10 16:30:36.037766 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.037743 2576 scope.go:117] "RemoveContainer" containerID="7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163" Jul 10 16:30:36.038082 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:30:36.038047 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163\": container with ID starting with 7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163 not found: ID does not exist" containerID="7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163" Jul 10 16:30:36.038180 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.038090 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163"} err="failed to get container status \"7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163\": rpc error: code = NotFound desc = could not find container \"7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163\": container with ID starting with 7800cd8770821c1d3c9e1c4fdaaaf45dcd57220b733ac9dfdecfaa9f5cb1b163 not found: ID does not exist" Jul 10 16:30:36.050278 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.050224 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck"] Jul 10 16:30:36.053738 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.053715 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-efe9d3-29728350-lgpck"] Jul 10 16:30:36.547143 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.547104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:36.768395 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.768352 2576 patch_prober.go:28] interesting pod/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 16:30:36.768582 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.768413 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 16:30:36.918555 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:36.918452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:37.105039 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:37.104995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:37.205595 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:37.205565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:37.423346 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:37.419747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:37.501313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:37.501199 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb3b483-da08-4907-a864-76d59dfe7544" path="/var/lib/kubelet/pods/bdb3b483-da08-4907-a864-76d59dfe7544/volumes" Jul 10 16:30:37.769022 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:37.768925 2576 patch_prober.go:28] interesting pod/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 16:30:37.769022 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:37.768983 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 16:30:38.771988 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:38.771959 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:38.772417 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:38.772297 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:38.773063 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:38.773049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:42.960544 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:42.960510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:43.062826 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:43.062791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:43.243492 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:43.243407 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww"] Jul 10 16:30:43.243641 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:43.243618 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" containerID="cri-o://2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628" gracePeriod=30 Jul 10 16:30:43.259384 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:43.259355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:43.454730 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:43.454697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:43.645591 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:43.645502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:44.099659 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.099625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:44.344939 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.344908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:44.446033 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.445997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:44.584658 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.584625 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp"] Jul 10 16:30:44.585125 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.585068 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" podUID="c5a56517-3924-4a64-9c3d-5c375d269ed7" containerName="maas-api" containerID="cri-o://81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb" gracePeriod=30 Jul 10 16:30:44.607730 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.607705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:44.850643 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.850612 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:44.928932 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.928894 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vvmb\" (UniqueName: \"kubernetes.io/projected/c5a56517-3924-4a64-9c3d-5c375d269ed7-kube-api-access-8vvmb\") pod \"c5a56517-3924-4a64-9c3d-5c375d269ed7\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " Jul 10 16:30:44.928932 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.928931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a56517-3924-4a64-9c3d-5c375d269ed7-openshift-service-ca\") pod \"c5a56517-3924-4a64-9c3d-5c375d269ed7\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " Jul 10 16:30:44.929189 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.929036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a56517-3924-4a64-9c3d-5c375d269ed7-maas-api-tls\") pod \"c5a56517-3924-4a64-9c3d-5c375d269ed7\" (UID: \"c5a56517-3924-4a64-9c3d-5c375d269ed7\") " Jul 10 16:30:44.929420 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.929393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a56517-3924-4a64-9c3d-5c375d269ed7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5a56517-3924-4a64-9c3d-5c375d269ed7" (UID: "c5a56517-3924-4a64-9c3d-5c375d269ed7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:30:44.931222 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.931187 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a56517-3924-4a64-9c3d-5c375d269ed7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5a56517-3924-4a64-9c3d-5c375d269ed7" (UID: "c5a56517-3924-4a64-9c3d-5c375d269ed7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:30:44.931412 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:44.931295 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a56517-3924-4a64-9c3d-5c375d269ed7-kube-api-access-8vvmb" (OuterVolumeSpecName: "kube-api-access-8vvmb") pod "c5a56517-3924-4a64-9c3d-5c375d269ed7" (UID: "c5a56517-3924-4a64-9c3d-5c375d269ed7"). InnerVolumeSpecName "kube-api-access-8vvmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:30:45.030494 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.030399 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a56517-3924-4a64-9c3d-5c375d269ed7-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:45.030494 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.030432 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvmb\" (UniqueName: \"kubernetes.io/projected/c5a56517-3924-4a64-9c3d-5c375d269ed7-kube-api-access-8vvmb\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:45.030494 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.030444 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a56517-3924-4a64-9c3d-5c375d269ed7-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:45.078844 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.078811 2576 generic.go:356] "Generic (PLEG): container finished" podID="c5a56517-3924-4a64-9c3d-5c375d269ed7" containerID="81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb" exitCode=0 Jul 10 16:30:45.079016 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.078868 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" Jul 10 16:30:45.079016 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.078900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" event={"ID":"c5a56517-3924-4a64-9c3d-5c375d269ed7","Type":"ContainerDied","Data":"81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb"} Jul 10 16:30:45.079016 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.078978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp" event={"ID":"c5a56517-3924-4a64-9c3d-5c375d269ed7","Type":"ContainerDied","Data":"46f52962fde202d175973e648798020bf20e773f6c08c59c321b0471426c4e96"} Jul 10 16:30:45.079016 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.079000 2576 scope.go:117] "RemoveContainer" containerID="81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb" Jul 10 16:30:45.089161 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.089137 2576 scope.go:117] "RemoveContainer" containerID="81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb" Jul 10 16:30:45.089463 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:30:45.089442 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb\": container with ID starting with 81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb not found: ID does not exist" containerID="81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb" Jul 10 16:30:45.089550 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.089477 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb"} err="failed to get container status \"81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb\": rpc error: code = NotFound desc = could not find container \"81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb\": container with ID starting with 81438ade7f275a56ce4826f23ae478e7688939d1787c3df16a592f31a45d32bb not found: ID does not exist" Jul 10 16:30:45.099733 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.099708 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp"] Jul 10 16:30:45.102995 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.102966 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1375a2-576646fb96-xn8bp"] Jul 10 16:30:45.495827 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:45.495795 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a56517-3924-4a64-9c3d-5c375d269ed7" path="/var/lib/kubelet/pods/c5a56517-3924-4a64-9c3d-5c375d269ed7/volumes" Jul 10 16:30:48.509900 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.509830 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:48.564860 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.564826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-envoy\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.564868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-credential-socket\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.564892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-socket\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.564923 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2dw7\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-kube-api-access-v2dw7\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.564961 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-podinfo\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.564991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istiod-ca-cert\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565006 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-certs\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565062 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-data\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565460 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565107 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-token\") pod \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\" (UID: \"fc002e2d-1dbc-4a36-89c5-85e3a062e534\") " Jul 10 16:30:48.565460 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:48.565460 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565213 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:48.565610 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565458 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:48.565610 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565535 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-credential-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.565610 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565556 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.565610 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.565570 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-workload-certs\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.566217 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.566189 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:30:48.566487 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.566462 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:48.567399 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.567378 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-kube-api-access-v2dw7" (OuterVolumeSpecName: "kube-api-access-v2dw7") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "kube-api-access-v2dw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:30:48.567488 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.567414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 16:30:48.567488 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.567441 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:30:48.567595 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.567559 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc002e2d-1dbc-4a36-89c5-85e3a062e534" (UID: "fc002e2d-1dbc-4a36-89c5-85e3a062e534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:30:48.666777 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.666744 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2dw7\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-kube-api-access-v2dw7\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.666777 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.666776 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-podinfo\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.667005 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.666788 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istiod-ca-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.667005 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.666801 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-data\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.667005 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.666812 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-token\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:48.667005 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:48.666823 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc002e2d-1dbc-4a36-89c5-85e3a062e534-istio-envoy\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:30:49.102590 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.102551 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerID="2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628" exitCode=0 Jul 10 16:30:49.102780 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.102604 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" Jul 10 16:30:49.102780 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.102617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" event={"ID":"fc002e2d-1dbc-4a36-89c5-85e3a062e534","Type":"ContainerDied","Data":"2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628"} Jul 10 16:30:49.102780 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.102669 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww" event={"ID":"fc002e2d-1dbc-4a36-89c5-85e3a062e534","Type":"ContainerDied","Data":"7afffc2372c5c73850133ba4076ed954331d5ad137505c706799f60d734dab1e"} Jul 10 16:30:49.102780 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.102695 2576 scope.go:117] "RemoveContainer" containerID="2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628" Jul 10 16:30:49.117503 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.113681 2576 scope.go:117] "RemoveContainer" containerID="7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb" Jul 10 16:30:49.124904 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.124874 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww"] Jul 10 16:30:49.126282 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.126226 2576 scope.go:117] "RemoveContainer" containerID="2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628" Jul 10 16:30:49.126633 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:30:49.126612 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628\": container with ID starting with 2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628 not found: ID does not exist" containerID="2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628" Jul 10 16:30:49.126710 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.126641 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628"} err="failed to get container status \"2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628\": rpc error: code = NotFound desc = could not find container \"2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628\": container with ID starting with 2a64acb506ef5c49f714a3cada6104d49dac6d660494de81df31ae5eda46f628 not found: ID does not exist" Jul 10 16:30:49.126710 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.126660 2576 scope.go:117] "RemoveContainer" containerID="7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb" Jul 10 16:30:49.126921 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:30:49.126897 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb\": container with ID starting with 7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb not found: ID does not exist" containerID="7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb" Jul 10 16:30:49.127080 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.126926 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb"} err="failed to get container status \"7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb\": rpc error: code = NotFound desc = could not find container \"7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb\": container with ID starting with 7c9cabc511aebfdf3789a892c6a8a38396ec756191d621bb726342272e7175bb not found: ID does not exist" Jul 10 16:30:49.129620 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.129597 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2b206a-openshift-default-865ffd4585-sqfww"] Jul 10 16:30:49.496225 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:49.496189 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" path="/var/lib/kubelet/pods/fc002e2d-1dbc-4a36-89c5-85e3a062e534/volumes" Jul 10 16:30:50.382863 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:50.382829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:50.486066 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:50.486035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:50.708110 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:50.708075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:50.899628 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:50.899582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:51.244033 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.244000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:51.534019 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.533929 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn"] Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534363 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a56517-3924-4a64-9c3d-5c375d269ed7" containerName="maas-api" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534375 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a56517-3924-4a64-9c3d-5c375d269ed7" containerName="maas-api" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534385 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3b483-da08-4907-a864-76d59dfe7544" containerName="cleanup" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534391 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3b483-da08-4907-a864-76d59dfe7544" containerName="cleanup" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534401 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534408 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534419 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" Jul 10 16:30:51.534421 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534426 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" Jul 10 16:30:51.534694 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534505 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" Jul 10 16:30:51.534694 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534516 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3b483-da08-4907-a864-76d59dfe7544" containerName="cleanup" Jul 10 16:30:51.534694 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534522 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc002e2d-1dbc-4a36-89c5-85e3a062e534" containerName="istio-proxy" Jul 10 16:30:51.534694 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.534528 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a56517-3924-4a64-9c3d-5c375d269ed7" containerName="maas-api" Jul 10 16:30:51.537260 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.537230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.539937 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.539922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c32d0d-openshift-default-dockercfg-wcfxp\"" Jul 10 16:30:51.551625 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.551602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn"] Jul 10 16:30:51.558459 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.558416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:51.689654 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.689623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:51.696795 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.696769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2l7w\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-kube-api-access-c2l7w\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.696924 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.696809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-credential-socket\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.696924 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.696874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-data\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.696924 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.696914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-token\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.697052 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.697031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-certs\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.697095 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.697050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-podinfo\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.697146 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.697095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istiod-ca-cert\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.697146 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.697126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-socket\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.697221 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.697158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-envoy\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798268 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-certs\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798268 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-podinfo\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798268 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istiod-ca-cert\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798268 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-socket\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798268 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-envoy\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798671 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2l7w\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-kube-api-access-c2l7w\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798671 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-credential-socket\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798671 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-data\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798671 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-token\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798671 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-certs\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798912 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-data\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798912 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-credential-socket\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.798912 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.798883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-socket\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.799042 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.799024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istiod-ca-cert\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.801247 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.801215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-envoy\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.801445 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.801430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-podinfo\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.809307 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.809280 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-token\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.809440 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.809401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2l7w\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-kube-api-access-c2l7w\") pod \"e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.850281 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.850219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:51.994706 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.994681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn"] Jul 10 16:30:51.996456 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:30:51.996431 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777ccbaa_4c60_43e7_9fa6_ce5220ef91dc.slice/crio-50987af22b8c35204a60526d7a23c30569adb690ac7fde24f5029604c76766b9 WatchSource:0}: Error finding container 50987af22b8c35204a60526d7a23c30569adb690ac7fde24f5029604c76766b9: Status 404 returned error can't find the container with id 50987af22b8c35204a60526d7a23c30569adb690ac7fde24f5029604c76766b9 Jul 10 16:30:51.998697 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.998653 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:51.998788 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.998729 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:51.998788 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:51.998768 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:30:52.121687 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:52.121645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" event={"ID":"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc","Type":"ContainerStarted","Data":"9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469"} Jul 10 16:30:52.121884 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:52.121714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" event={"ID":"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc","Type":"ContainerStarted","Data":"50987af22b8c35204a60526d7a23c30569adb690ac7fde24f5029604c76766b9"} Jul 10 16:30:52.145360 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:52.145303 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podStartSLOduration=1.145282647 podStartE2EDuration="1.145282647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:30:52.145212309 +0000 UTC m=+2141.216030403" watchObservedRunningTime="2026-07-10 16:30:52.145282647 +0000 UTC m=+2141.216100718" Jul 10 16:30:52.851130 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:52.851086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:53.855620 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:53.855589 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:54.131796 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.131712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:54.132769 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.132750 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:30:54.214402 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.214366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:54.302545 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.302509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:54.317587 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.317550 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld"] Jul 10 16:30:54.320533 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.320512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.322509 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.322491 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c32d0d-serving-cert\"" Jul 10 16:30:54.331323 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.331301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld"] Jul 10 16:30:54.401231 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.401146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:54.424922 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.424883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c0cc03-1fd2-44bc-b20d-a97127333961-maas-api-tls\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.425100 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.425027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ph56\" (UniqueName: \"kubernetes.io/projected/55c0cc03-1fd2-44bc-b20d-a97127333961-kube-api-access-9ph56\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.425100 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.425066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c0cc03-1fd2-44bc-b20d-a97127333961-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.501502 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.501459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:54.525575 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.525532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c0cc03-1fd2-44bc-b20d-a97127333961-maas-api-tls\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.525745 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.525627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ph56\" (UniqueName: \"kubernetes.io/projected/55c0cc03-1fd2-44bc-b20d-a97127333961-kube-api-access-9ph56\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.525745 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.525659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c0cc03-1fd2-44bc-b20d-a97127333961-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.526396 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.526373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c0cc03-1fd2-44bc-b20d-a97127333961-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.528312 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.528290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c0cc03-1fd2-44bc-b20d-a97127333961-maas-api-tls\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.537521 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.537498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ph56\" (UniqueName: \"kubernetes.io/projected/55c0cc03-1fd2-44bc-b20d-a97127333961-kube-api-access-9ph56\") pod \"maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.632311 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.632263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:54.779533 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.779506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld"] Jul 10 16:30:54.782740 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:30:54.782698 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c0cc03_1fd2_44bc_b20d_a97127333961.slice/crio-8eea59e9fd7c2b494026806f012edba02de1209e4244b44bcdac3c3548330dbc WatchSource:0}: Error finding container 8eea59e9fd7c2b494026806f012edba02de1209e4244b44bcdac3c3548330dbc: Status 404 returned error can't find the container with id 8eea59e9fd7c2b494026806f012edba02de1209e4244b44bcdac3c3548330dbc Jul 10 16:30:54.942547 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:54.942471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:55.046153 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:55.046105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:55.137191 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:55.137161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" event={"ID":"55c0cc03-1fd2-44bc-b20d-a97127333961","Type":"ContainerStarted","Data":"8eea59e9fd7c2b494026806f012edba02de1209e4244b44bcdac3c3548330dbc"} Jul 10 16:30:56.143341 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:56.143306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" event={"ID":"55c0cc03-1fd2-44bc-b20d-a97127333961","Type":"ContainerStarted","Data":"1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da"} Jul 10 16:30:56.143780 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:56.143516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:30:56.165203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:56.165157 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" podStartSLOduration=1.818194439 podStartE2EDuration="2.165140967s" totalImagesPullingTime="346.946528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:30:56.164523472 +0000 UTC m=+2145.235341543" watchObservedRunningTime="2026-07-10 16:30:56.165140967 +0000 UTC m=+2145.235959038" Jul 10 16:30:58.620778 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:58.620734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:58.953541 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:58.953504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:59.299224 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:59.299102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:30:59.555410 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:30:59.555316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:02.152420 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:02.152323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:02.155657 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:02.155634 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:31:02.283809 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:02.283777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:02.431943 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:02.431863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:02.650740 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:02.650706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:03.176991 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:03.176955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:03.259212 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:03.259176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:06.587691 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:06.587648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:09.174517 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:09.174472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:10.173448 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:10.173408 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 16:31:10.173644 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:10.173471 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 16:31:11.230847 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:11.230761 2576 generic.go:356] "Generic (PLEG): container finished" podID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerID="9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469" exitCode=0 Jul 10 16:31:11.231289 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:11.230835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" event={"ID":"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc","Type":"ContainerDied","Data":"9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469"} Jul 10 16:31:11.231289 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:11.231215 2576 scope.go:117] "RemoveContainer" containerID="9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469" Jul 10 16:31:11.232313 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:11.232288 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:31:11.232375 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:11.232354 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:31:11.232415 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:11.232405 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 16:31:12.237325 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:12.237292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" event={"ID":"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc","Type":"ContainerStarted","Data":"69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342"} Jul 10 16:31:12.850946 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:12.850907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:31:12.851186 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:12.851162 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 16:31:12.851326 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:12.851221 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 16:31:13.851218 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:13.851181 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 16:31:13.851662 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:13.851240 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 16:31:14.851510 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:14.851464 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 16:31:14.851897 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:14.851529 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 16:31:15.851527 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:15.851484 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 16:31:15.852026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:15.851544 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 16:31:15.869480 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:15.869446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:15.968712 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:15.968678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:16.854339 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:16.854313 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:31:16.854770 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:16.854572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:31:16.855433 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:16.855412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:31:18.582364 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:18.582329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:25.436035 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:25.435999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:25.535227 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:25.535191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:31:39.838156 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:31:39.838117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:32:21.143491 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:32:21.143439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:33:43.594243 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:33:43.594207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:33:45.204477 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:33:45.204445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:33:55.757731 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:33:55.757697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:33:59.881410 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:33:59.881377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:34:32.540721 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:34:32.540629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:11.552708 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:11.552679 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:35:11.555954 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:11.555932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:35:11.556725 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:11.556702 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:35:11.559394 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:11.559376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ssxhb_0e9aeccc-b117-4783-8c2d-09c138f839e6/ovn-acl-logging/0.log" Jul 10 16:35:16.057876 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:16.057845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:17.041506 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:17.041466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:17.163796 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:17.163766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:17.265026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:17.264995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:23.636762 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:23.636730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:34.664465 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:34.664386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:35:47.458228 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:35:47.458196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:16.669939 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:16.669898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:16.772518 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:16.772478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:26.412233 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:26.412194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:26.513836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:26.513798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:38.432894 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:38.432859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:39.446486 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:39.446448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:58.533899 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:58.533858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:58.994264 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:58.994200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:59.095562 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.095514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:59.220410 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.220370 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld"] Jul 10 16:36:59.221019 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.220985 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" podUID="55c0cc03-1fd2-44bc-b20d-a97127333961" containerName="maas-api" containerID="cri-o://1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da" gracePeriod=30 Jul 10 16:36:59.241626 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.241601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:36:59.476748 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.476721 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:36:59.536954 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.536919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ph56\" (UniqueName: \"kubernetes.io/projected/55c0cc03-1fd2-44bc-b20d-a97127333961-kube-api-access-9ph56\") pod \"55c0cc03-1fd2-44bc-b20d-a97127333961\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " Jul 10 16:36:59.537406 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.537008 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c0cc03-1fd2-44bc-b20d-a97127333961-openshift-service-ca\") pod \"55c0cc03-1fd2-44bc-b20d-a97127333961\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " Jul 10 16:36:59.537406 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.537055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c0cc03-1fd2-44bc-b20d-a97127333961-maas-api-tls\") pod \"55c0cc03-1fd2-44bc-b20d-a97127333961\" (UID: \"55c0cc03-1fd2-44bc-b20d-a97127333961\") " Jul 10 16:36:59.537534 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.537404 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c0cc03-1fd2-44bc-b20d-a97127333961-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55c0cc03-1fd2-44bc-b20d-a97127333961" (UID: "55c0cc03-1fd2-44bc-b20d-a97127333961"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:36:59.539420 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.539388 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c0cc03-1fd2-44bc-b20d-a97127333961-kube-api-access-9ph56" (OuterVolumeSpecName: "kube-api-access-9ph56") pod "55c0cc03-1fd2-44bc-b20d-a97127333961" (UID: "55c0cc03-1fd2-44bc-b20d-a97127333961"). InnerVolumeSpecName "kube-api-access-9ph56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:36:59.539420 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.539414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c0cc03-1fd2-44bc-b20d-a97127333961-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55c0cc03-1fd2-44bc-b20d-a97127333961" (UID: "55c0cc03-1fd2-44bc-b20d-a97127333961"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:36:59.638356 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.638107 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ph56\" (UniqueName: \"kubernetes.io/projected/55c0cc03-1fd2-44bc-b20d-a97127333961-kube-api-access-9ph56\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:36:59.638356 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.638150 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c0cc03-1fd2-44bc-b20d-a97127333961-openshift-service-ca\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:36:59.638356 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:36:59.638166 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c0cc03-1fd2-44bc-b20d-a97127333961-maas-api-tls\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:00.033646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.033610 2576 generic.go:356] "Generic (PLEG): container finished" podID="55c0cc03-1fd2-44bc-b20d-a97127333961" containerID="1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da" exitCode=0 Jul 10 16:37:00.033836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.033661 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" Jul 10 16:37:00.033836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.033691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" event={"ID":"55c0cc03-1fd2-44bc-b20d-a97127333961","Type":"ContainerDied","Data":"1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da"} Jul 10 16:37:00.033836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.033752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld" event={"ID":"55c0cc03-1fd2-44bc-b20d-a97127333961","Type":"ContainerDied","Data":"8eea59e9fd7c2b494026806f012edba02de1209e4244b44bcdac3c3548330dbc"} Jul 10 16:37:00.033836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.033783 2576 scope.go:117] "RemoveContainer" containerID="1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da" Jul 10 16:37:00.043467 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.043442 2576 scope.go:117] "RemoveContainer" containerID="1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da" Jul 10 16:37:00.043746 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:37:00.043726 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da\": container with ID starting with 1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da not found: ID does not exist" containerID="1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da" Jul 10 16:37:00.043818 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.043761 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da"} err="failed to get container status \"1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da\": rpc error: code = NotFound desc = could not find container \"1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da\": container with ID starting with 1ac36db7100d19ee20bc137aa05ac4407badf6536b63bf814057d16ad3b5b6da not found: ID does not exist" Jul 10 16:37:00.055567 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.055542 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld"] Jul 10 16:37:00.059512 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:00.059488 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c32d0d-7975b4f4c5-p57ld"] Jul 10 16:37:01.496571 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:01.496541 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c0cc03-1fd2-44bc-b20d-a97127333961" path="/var/lib/kubelet/pods/55c0cc03-1fd2-44bc-b20d-a97127333961/volumes" Jul 10 16:37:05.241213 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.241162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:05.343527 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.343492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:05.545525 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.545440 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn"] Jul 10 16:37:05.545669 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.545650 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" containerID="cri-o://69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342" gracePeriod=30 Jul 10 16:37:05.556812 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.556776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:05.766455 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.766422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:05.940096 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:05.940070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:06.414636 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:06.414519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:06.515444 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:06.515410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:06.617646 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:06.617609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:06.683778 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:06.683750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:10.804633 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.804605 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:37:10.839131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839037 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istiod-ca-cert\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-envoy\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839131 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839116 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-socket\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839173 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-credential-socket\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839231 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2l7w\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-kube-api-access-c2l7w\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839289 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-data\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839321 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-certs\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839355 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-token\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839454 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-podinfo\") pod \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\" (UID: \"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc\") " Jul 10 16:37:10.839763 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839434 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:37:10.839763 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839536 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:37:10.839763 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839715 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istiod-ca-cert\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.839763 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839736 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.839763 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839739 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:37:10.840022 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.839853 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:37:10.840502 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.840381 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:37:10.841637 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.841614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:37:10.841943 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.841913 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:37:10.842059 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.841998 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 16:37:10.842179 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.842156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-kube-api-access-c2l7w" (OuterVolumeSpecName: "kube-api-access-c2l7w") pod "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" (UID: "777ccbaa-4c60-43e7-9fa6-ce5220ef91dc"). InnerVolumeSpecName "kube-api-access-c2l7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:37:10.940555 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940514 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-envoy\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.940555 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940549 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-credential-socket\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.940555 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940559 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2l7w\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-kube-api-access-c2l7w\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.940555 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940568 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-data\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.940836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940576 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-workload-certs\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.940836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940583 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-token\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:10.940836 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:10.940590 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc-istio-podinfo\") on node \"ip-10-0-139-69.ec2.internal\" DevicePath \"\"" Jul 10 16:37:11.096461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.096359 2576 generic.go:356] "Generic (PLEG): container finished" podID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerID="69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342" exitCode=0 Jul 10 16:37:11.096461 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.096416 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" Jul 10 16:37:11.096692 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.096418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" event={"ID":"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc","Type":"ContainerDied","Data":"69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342"} Jul 10 16:37:11.096692 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.096536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn" event={"ID":"777ccbaa-4c60-43e7-9fa6-ce5220ef91dc","Type":"ContainerDied","Data":"50987af22b8c35204a60526d7a23c30569adb690ac7fde24f5029604c76766b9"} Jul 10 16:37:11.096692 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.096558 2576 scope.go:117] "RemoveContainer" containerID="69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342" Jul 10 16:37:11.106332 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.106315 2576 scope.go:117] "RemoveContainer" containerID="9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469" Jul 10 16:37:11.114989 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.114967 2576 scope.go:117] "RemoveContainer" containerID="69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342" Jul 10 16:37:11.115300 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:37:11.115272 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342\": container with ID starting with 69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342 not found: ID does not exist" containerID="69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342" Jul 10 16:37:11.115392 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.115307 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342"} err="failed to get container status \"69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342\": rpc error: code = NotFound desc = could not find container \"69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342\": container with ID starting with 69d6be8283851c257f1f198ceaa62ce04e07edf7f20c5b3bd7d84cebc0e1a342 not found: ID does not exist" Jul 10 16:37:11.115392 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.115324 2576 scope.go:117] "RemoveContainer" containerID="9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469" Jul 10 16:37:11.115591 ip-10-0-139-69 kubenswrapper[2576]: E0710 16:37:11.115569 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469\": container with ID starting with 9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469 not found: ID does not exist" containerID="9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469" Jul 10 16:37:11.115653 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.115600 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469"} err="failed to get container status \"9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469\": rpc error: code = NotFound desc = could not find container \"9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469\": container with ID starting with 9ba6c998bb4b7935623671c5584fef4f592b4cba998175ef33da7e0092abc469 not found: ID does not exist" Jul 10 16:37:11.118270 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.118225 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn"] Jul 10 16:37:11.128680 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.128657 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c32d0d-openshift-default-f84d6bdd7-9qnfn"] Jul 10 16:37:11.495818 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:11.495782 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" path="/var/lib/kubelet/pods/777ccbaa-4c60-43e7-9fa6-ce5220ef91dc/volumes" Jul 10 16:37:12.533656 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:12.533616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:12.633929 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:12.633892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:12.806919 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:12.806822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:13.021707 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:13.021672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:18.497181 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:18.497154 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4nlh7_bc08ecc2-5b93-4b46-8319-923bc7c4f4e7/manager/0.log" Jul 10 16:37:18.774209 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:18.774113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787666694b-8jkpg_42ac1655-2931-4170-a3d8-ccc9f1bd198c/manager/0.log" Jul 10 16:37:19.007943 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:19.007909 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v8ffm_0dbcfb7c-695b-471f-b775-79fbeccb1d59/manager/0.log" Jul 10 16:37:19.384232 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:19.384200 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j_ea0f5676-7790-41fb-883d-55bf43d1d7f1/manager/0.log" Jul 10 16:37:19.398930 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:19.398898 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j_ea0f5676-7790-41fb-883d-55bf43d1d7f1/copy-manifests/0.log" Jul 10 16:37:19.742853 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:19.742823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c4df6cbf-bdw4k_069d269e-1902-44a7-9c7b-96efca165806/maas-api/0.log" Jul 10 16:37:19.863609 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:19.863581 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-dk5f6_74fa0c47-0dcc-444e-9b6a-532887254c61/postgres/0.log" Jul 10 16:37:21.421381 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:21.421344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rrpnc_f81f11a3-e278-4b28-a195-3c89327c8466/manager/0.log" Jul 10 16:37:21.649276 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:21.649219 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-p7pnr_bef8df15-9934-424c-967f-8bef761df49e/registry-server/0.log" Jul 10 16:37:21.779208 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:21.779180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fs64b_062169f4-e0bc-4584-8aa9-26253f1ef984/manager/0.log" Jul 10 16:37:22.509246 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:22.509212 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hpm78_c5c36812-f887-4b3a-acd0-c77dbd9c12dc/discovery/0.log" Jul 10 16:37:22.623818 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:22.623790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8457fcdff5-k6w64_2714aff7-5112-4a2a-8a79-7fc66d6c52d3/kube-auth-proxy/0.log" Jul 10 16:37:22.863727 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:22.863639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/1.log" Jul 10 16:37:22.985530 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:22.985497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fq6hb_51cd78d6-3169-425c-b3c1-7454f496c253/payload-pre-processing/0.log" Jul 10 16:37:23.105551 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:23.105520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9v2vw_f6e3f4b2-f74d-43d6-b21c-48d250837725/payload-processing/0.log" Jul 10 16:37:23.575026 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:23.574999 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-pvljd_279230e9-3502-4c02-ac04-52fe7c0441cb/main/0.log" Jul 10 16:37:23.798177 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:23.798146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-fhsmk_97d58833-9980-48cc-887b-766a0b86044d/main/0.log" Jul 10 16:37:24.031052 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:24.031024 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6bbrz_bfcb0dec-a8e4-43b7-ad9b-06cdd697ea9e/main/0.log" Jul 10 16:37:28.583610 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:28.583576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:28.681081 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:28.681048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:31.882884 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:31.882857 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kw467_58620b37-9b5a-40fb-b71b-d2c06e229147/global-pull-secret-syncer/0.log" Jul 10 16:37:32.034801 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:32.034769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" Jul 10 16:37:32.195917 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:32.195889 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-69.ec2.internal_af23fa6bfd969eb2ea8a7d64ea7106f7/haproxy/0.log" Jul 10 16:37:37.813944 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:37.813914 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rrpnc_f81f11a3-e278-4b28-a195-3c89327c8466/manager/0.log" Jul 10 16:37:37.879645 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:37.879614 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-p7pnr_bef8df15-9934-424c-967f-8bef761df49e/registry-server/0.log" Jul 10 16:37:37.974649 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:37.974618 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fs64b_062169f4-e0bc-4584-8aa9-26253f1ef984/manager/0.log" Jul 10 16:37:39.909616 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:39.909587 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-wk966_573e9a41-ff70-4e11-a4f4-aa94e16e6183/cluster-monitoring-operator/0.log" Jul 10 16:37:40.369572 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.369541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ztlh4_fc9abbc1-7d87-453f-a05f-f3147681ca3f/node-exporter/0.log" Jul 10 16:37:40.421111 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.421084 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ztlh4_fc9abbc1-7d87-453f-a05f-f3147681ca3f/kube-rbac-proxy/0.log" Jul 10 16:37:40.485711 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.485686 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ztlh4_fc9abbc1-7d87-453f-a05f-f3147681ca3f/init-textfile/0.log" Jul 10 16:37:40.548302 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.548275 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-vhhfx_d9afae43-1a88-4555-a8e2-7d4148f0f0d8/kube-rbac-proxy-main/0.log" Jul 10 16:37:40.577143 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.577116 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-vhhfx_d9afae43-1a88-4555-a8e2-7d4148f0f0d8/kube-rbac-proxy-self/0.log" Jul 10 16:37:40.599275 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.599230 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-vhhfx_d9afae43-1a88-4555-a8e2-7d4148f0f0d8/openshift-state-metrics/0.log" Jul 10 16:37:40.974184 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.974153 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58b7fb7f8f-f927w_82924150-518a-4279-91a3-04dc86dd1785/thanos-query/0.log" Jul 10 16:37:40.995595 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:40.995563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58b7fb7f8f-f927w_82924150-518a-4279-91a3-04dc86dd1785/kube-rbac-proxy-web/0.log" Jul 10 16:37:41.018239 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.018215 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58b7fb7f8f-f927w_82924150-518a-4279-91a3-04dc86dd1785/kube-rbac-proxy/0.log" Jul 10 16:37:41.041491 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.041467 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58b7fb7f8f-f927w_82924150-518a-4279-91a3-04dc86dd1785/prom-label-proxy/0.log" Jul 10 16:37:41.061664 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.061631 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58b7fb7f8f-f927w_82924150-518a-4279-91a3-04dc86dd1785/kube-rbac-proxy-rules/0.log" Jul 10 16:37:41.101398 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.101374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58b7fb7f8f-f927w_82924150-518a-4279-91a3-04dc86dd1785/kube-rbac-proxy-metrics/0.log" Jul 10 16:37:41.922417 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.922388 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr"] Jul 10 16:37:41.922862 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.922844 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" Jul 10 16:37:41.922960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.922865 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" Jul 10 16:37:41.922960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.922884 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0cc03-1fd2-44bc-b20d-a97127333961" containerName="maas-api" Jul 10 16:37:41.922960 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.922892 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0cc03-1fd2-44bc-b20d-a97127333961" containerName="maas-api" Jul 10 16:37:41.923110 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.922997 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c0cc03-1fd2-44bc-b20d-a97127333961" containerName="maas-api" Jul 10 16:37:41.923110 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.923015 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" Jul 10 16:37:41.923110 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.923024 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="777ccbaa-4c60-43e7-9fa6-ce5220ef91dc" containerName="istio-proxy" Jul 10 16:37:41.926527 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.926505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:41.928736 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.928714 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wrzbj\"/\"kube-root-ca.crt\"" Jul 10 16:37:41.928859 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.928792 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wrzbj\"/\"openshift-service-ca.crt\"" Jul 10 16:37:41.929067 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.929054 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wrzbj\"/\"default-dockercfg-bbxt2\"" Jul 10 16:37:41.932538 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:41.932478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr"] Jul 10 16:37:42.023765 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.023721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-podres\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.024203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.023805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfqt\" (UniqueName: \"kubernetes.io/projected/27f65491-9a9b-4bba-b144-8b0d40091889-kube-api-access-whfqt\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.024203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.023841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-sys\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.024203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.023878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-proc\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.024203 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.023911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-lib-modules\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124464 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfqt\" (UniqueName: \"kubernetes.io/projected/27f65491-9a9b-4bba-b144-8b0d40091889-kube-api-access-whfqt\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-sys\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-proc\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-lib-modules\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-sys\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-podres\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124670 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-proc\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124997 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-lib-modules\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.124997 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.124769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27f65491-9a9b-4bba-b144-8b0d40091889-podres\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.134941 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.134911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfqt\" (UniqueName: \"kubernetes.io/projected/27f65491-9a9b-4bba-b144-8b0d40091889-kube-api-access-whfqt\") pod \"perf-node-gather-daemonset-2wqtr\" (UID: \"27f65491-9a9b-4bba-b144-8b0d40091889\") " pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.238265 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.238226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:42.377340 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.377310 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr"] Jul 10 16:37:42.379837 ip-10-0-139-69 kubenswrapper[2576]: W0710 16:37:42.379805 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27f65491_9a9b_4bba_b144_8b0d40091889.slice/crio-7ad1f6826ee7c5e55f79d5f7a30a0af27789ba883f247ca93dc3200504611162 WatchSource:0}: Error finding container 7ad1f6826ee7c5e55f79d5f7a30a0af27789ba883f247ca93dc3200504611162: Status 404 returned error can't find the container with id 7ad1f6826ee7c5e55f79d5f7a30a0af27789ba883f247ca93dc3200504611162 Jul 10 16:37:42.889444 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.889419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/1.log" Jul 10 16:37:42.894846 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:42.894812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hzdr6_ef6f967d-0657-47e5-bb08-efaffbaca4fa/console-operator/2.log" Jul 10 16:37:43.256798 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:43.256765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" event={"ID":"27f65491-9a9b-4bba-b144-8b0d40091889","Type":"ContainerStarted","Data":"35b71e4c801c354c692082870c1b634d164e3aa10acf142f4567760cc5ca5819"} Jul 10 16:37:43.257307 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:43.256817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" event={"ID":"27f65491-9a9b-4bba-b144-8b0d40091889","Type":"ContainerStarted","Data":"7ad1f6826ee7c5e55f79d5f7a30a0af27789ba883f247ca93dc3200504611162"} Jul 10 16:37:43.257307 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:43.256839 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:43.274675 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:43.274626 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" podStartSLOduration=2.27460969 podStartE2EDuration="2.27460969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:43.273477413 +0000 UTC m=+2552.344295486" watchObservedRunningTime="2026-07-10 16:37:43.27460969 +0000 UTC m=+2552.345427761" Jul 10 16:37:45.092468 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:45.092434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvmrt_5b65bd4f-cccb-4ae4-929d-e1f2331394a8/dns/0.log" Jul 10 16:37:45.120876 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:45.120847 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvmrt_5b65bd4f-cccb-4ae4-929d-e1f2331394a8/kube-rbac-proxy/0.log" Jul 10 16:37:45.280566 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:45.280540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bwg7j_fc4cbeda-9ded-46f4-96fd-3222264a0097/dns-node-resolver/0.log" Jul 10 16:37:45.898757 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:45.898730 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-28vk6_97dbbccd-be38-4062-ac31-8ea70e6e6991/node-ca/0.log" Jul 10 16:37:47.035831 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.035802 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hpm78_c5c36812-f887-4b3a-acd0-c77dbd9c12dc/discovery/0.log" Jul 10 16:37:47.060898 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.060872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8457fcdff5-k6w64_2714aff7-5112-4a2a-8a79-7fc66d6c52d3/kube-auth-proxy/0.log" Jul 10 16:37:47.122896 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.122864 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/0.log" Jul 10 16:37:47.150486 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.150455 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/1.log" Jul 10 16:37:47.180509 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.180483 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fq6hb_51cd78d6-3169-425c-b3c1-7454f496c253/payload-pre-processing/0.log" Jul 10 16:37:47.213151 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.213115 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9v2vw_f6e3f4b2-f74d-43d6-b21c-48d250837725/payload-processing/0.log" Jul 10 16:37:47.956954 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:47.956922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8z8w6_db137799-74b2-45af-9d93-b93d5939ffdb/serve-healthcheck-canary/0.log" Jul 10 16:37:48.540629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:48.540598 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55czr_5ccde3d4-bbf9-44ac-a89c-f36b83af6db2/kube-rbac-proxy/0.log" Jul 10 16:37:48.562080 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:48.562052 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55czr_5ccde3d4-bbf9-44ac-a89c-f36b83af6db2/exporter/0.log" Jul 10 16:37:48.585304 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:48.585278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55czr_5ccde3d4-bbf9-44ac-a89c-f36b83af6db2/extractor/0.log" Jul 10 16:37:49.272425 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:49.272399 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wrzbj/perf-node-gather-daemonset-2wqtr" Jul 10 16:37:51.190004 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:51.189972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4nlh7_bc08ecc2-5b93-4b46-8319-923bc7c4f4e7/manager/0.log" Jul 10 16:37:51.536019 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:51.535987 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787666694b-8jkpg_42ac1655-2931-4170-a3d8-ccc9f1bd198c/manager/0.log" Jul 10 16:37:51.632764 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:51.632734 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v8ffm_0dbcfb7c-695b-471f-b775-79fbeccb1d59/manager/0.log" Jul 10 16:37:51.824408 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:51.824324 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j_ea0f5676-7790-41fb-883d-55bf43d1d7f1/manager/0.log" Jul 10 16:37:51.842657 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:51.842626 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c56c8c4bf-vfp9j_ea0f5676-7790-41fb-883d-55bf43d1d7f1/copy-manifests/0.log" Jul 10 16:37:52.623053 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:52.623008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:52.687208 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:52.687177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g9mg9_a5080f75-77f9-4f40-9a90-f8a5cbd4bc96/konnectivity-agent/0.log" limit=1024 Jul 10 16:37:52.921289 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:52.921187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hpm78_c5c36812-f887-4b3a-acd0-c77dbd9c12dc/discovery/0.log" Jul 10 16:37:52.932629 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:52.932603 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8457fcdff5-k6w64_2714aff7-5112-4a2a-8a79-7fc66d6c52d3/kube-auth-proxy/0.log" Jul 10 16:37:52.966826 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:52.966793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/0.log" Jul 10 16:37:52.995778 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:52.995745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/1.log" Jul 10 16:37:53.021976 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:53.021950 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fq6hb_51cd78d6-3169-425c-b3c1-7454f496c253/payload-pre-processing/0.log" Jul 10 16:37:53.044124 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:53.044097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9v2vw_f6e3f4b2-f74d-43d6-b21c-48d250837725/payload-processing/0.log" Jul 10 16:37:54.393179 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:54.393149 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hpm78_c5c36812-f887-4b3a-acd0-c77dbd9c12dc/discovery/0.log" Jul 10 16:37:54.406926 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:54.406898 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8457fcdff5-k6w64_2714aff7-5112-4a2a-8a79-7fc66d6c52d3/kube-auth-proxy/0.log" Jul 10 16:37:54.453536 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:54.453504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/0.log" Jul 10 16:37:54.483039 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:54.483010 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q7p9j_7a6c8ffd-3477-4bd1-8416-a12ff141e3af/istio-proxy/1.log" Jul 10 16:37:54.506355 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:54.506321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fq6hb_51cd78d6-3169-425c-b3c1-7454f496c253/payload-pre-processing/0.log" Jul 10 16:37:54.529435 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:54.529403 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9v2vw_f6e3f4b2-f74d-43d6-b21c-48d250837725/payload-processing/0.log" Jul 10 16:37:58.437366 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:58.437334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gr24d_2ff4b492-1992-40f8-992d-5d5d711ea24a/migrator/0.log" Jul 10 16:37:58.458439 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:58.458409 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gr24d_2ff4b492-1992-40f8-992d-5d5d711ea24a/graceful-termination/0.log" Jul 10 16:37:58.862775 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:58.862696 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-7dtpv_a95360d0-7c80-4f54-a510-47b017ebc24e/kube-storage-version-migrator-operator/1.log" Jul 10 16:37:58.863657 ip-10-0-139-69 kubenswrapper[2576]: I0710 16:37:58.863632 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-7dtpv_a95360d0-7c80-4f54-a510-47b017ebc24e/kube-storage-version-migrator-operator/0.log"